update p2
This commit is contained in:
mhjensen
@@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Project 2 on Machine Learning, deadline November 7">
|
||||
<meta name="description" content="Project 2 on Machine Learning, deadline November 9">
|
||||
|
||||
<title>Project 2 on Machine Learning, deadline November 7</title>
|
||||
<title>Project 2 on Machine Learning, deadline November 9</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
@@ -46,33 +46,38 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Part a): Write your Logistic Regression code, first step',
|
||||
('Part a): Write your own Stochastic Gradient Descent code, '
|
||||
'first step',
|
||||
3,
|
||||
None,
|
||||
'___sec1'),
|
||||
('Part b): Write your Logistic Regression code, second step',
|
||||
('Part b): Writing your own Neural Network code',
|
||||
3,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Part c): Writing your own Neural Network code',
|
||||
('Part c): Testing different activation functions',
|
||||
3,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Part d): Regression analysis using neural networks',
|
||||
('Part d): Classification analysis using neural networks',
|
||||
3,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Part e) Critical evaluation of the various algorithms',
|
||||
('Part e): Write your Logistic Regression code, final step',
|
||||
3,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Background literature', 2, None, '___sec6'),
|
||||
('Introduction to numerical projects', 2, None, '___sec7'),
|
||||
('Part f) Critical evaluation of the various algorithms',
|
||||
3,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Background literature', 2, None, '___sec7'),
|
||||
('Introduction to numerical projects', 2, None, '___sec8'),
|
||||
('Format for electronic delivery of report and programs',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Software and needed installations', 2, None, '___sec9')]}
|
||||
'___sec9'),
|
||||
('Software and needed installations', 2, None, '___sec10')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -102,7 +107,7 @@ MathJax.Hub.Config({
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Project2-bs.html">Project 2 on Machine Learning, deadline November 7</a>
|
||||
<a class="navbar-brand" href="Project2-bs.html">Project 2 on Machine Learning, deadline November 9</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
@@ -111,15 +116,16 @@ MathJax.Hub.Config({
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;"><b>Classification and Regression, from linear and logistic regression to neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;"> Part a): Write your Logistic Regression code, first step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;"> Part b): Write your Logistic Regression code, second step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;"> Part c): Writing your own Neural Network code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;"> Part d): Regression analysis using neural networks</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;"> Part e) Critical evaluation of the various algorithms</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;"><b>Background literature</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;"><b>Introduction to numerical projects</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;"><b>Format for electronic delivery of report and programs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;"><b>Software and needed installations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;"> Part a): Write your own Stochastic Gradient Descent code, first step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;"> Part b): Writing your own Neural Network code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;"> Part c): Testing different activation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;"> Part d): Classification analysis using neural networks</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;"> Part e): Write your Logistic Regression code, final step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;"> Part f) Critical evaluation of the various algorithms</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;"><b>Background literature</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;"><b>Introduction to numerical projects</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;"><b>Format for electronic delivery of report and programs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;"><b>Software and needed installations</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -138,7 +144,7 @@ MathJax.Hub.Config({
|
||||
|
||||
|
||||
<div class="jumbotron">
|
||||
<center><h1>Project 2 on Machine Learning, deadline November 7</h1></center> <!-- document title -->
|
||||
<center><h1>Project 2 on Machine Learning, deadline November 9</h1></center> <!-- document title -->
|
||||
|
||||
<p>
|
||||
<!-- author(s): <a href="http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" target="_self">Data Analysis and Machine Learning FYS-STK3155/FYS4155</a> -->
|
||||
@@ -153,7 +159,7 @@ MathJax.Hub.Config({
|
||||
<center><b>Department of Physics, University of Oslo, Norway</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 6, 2020</h4></center> <!-- date -->
|
||||
<center><h4>Oct 7, 2020</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
</div> <!-- end jumbotron -->
|
||||
@@ -163,8 +169,8 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
The main aim of this project is to study both classification and
|
||||
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
|
||||
in project 1. We will include logistic regresion for classification
|
||||
problems and write our own FFNNcode for studying
|
||||
in project 1. We will also include logistic regression for classification
|
||||
problems and write our own FFNN code for studying
|
||||
both regression and classification problems. The codes developed in
|
||||
project 1, including bootstrap and/or cross-validation as well as the
|
||||
computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can
|
||||
@@ -174,64 +180,111 @@ also be utilized in the present analysis.
|
||||
The data sets that we propose here are (the default sets)
|
||||
|
||||
<ul>
|
||||
<li> Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be
|
||||
<li> Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
|
||||
|
||||
<ol type="a"></li>
|
||||
<li> Either the Franke function or the terrain data from project 1, or data sets your propose.</li>
|
||||
</ol>
|
||||
|
||||
<li> Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine.</li>
|
||||
<li> Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called <a href="https://en.wikipedia.org/wiki/MNIST_database" target="_self">MNIST</a> data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_self">week 41</a></li>
|
||||
</ul>
|
||||
|
||||
However, if you would like to study other data sets, feel free to
|
||||
propose other sets. What we listed here are mere suggestions from our
|
||||
side. If you opt for another data set, consider using a set which
|
||||
has been studied in the scientific literature. This makes it easier
|
||||
for you to compare and analyze your results. It is also an essential
|
||||
elements of the scientific discussion.
|
||||
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
|
||||
element of the scientific discussion.
|
||||
|
||||
<p>
|
||||
In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_self">Scikit-Learn site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_self">University of California at Irvine</a>. The <a href="https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html" target="_self">lecture slides on dimensionality reduction have several code examples on this data set</a>.
|
||||
|
||||
<h3 id="___sec1" class="anchor">Part a): Write your Logistic Regression code, first step </h3>
|
||||
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_self">Scikit-Learn site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_self">University of California at Irvine</a>.
|
||||
|
||||
<p>
|
||||
If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article.
|
||||
We recommend also that you study the code example in the <a href="https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html" target="_self">Logistic Regression</a>.
|
||||
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
|
||||
|
||||
<h3 id="___sec1" class="anchor">Part a): Write your own Stochastic Gradient Descent code, first step </h3>
|
||||
|
||||
<p>
|
||||
Write the part of the code which reads in the data and sets up the relevant data sets.
|
||||
|
||||
<h3 id="___sec2" class="anchor">Part b): Write your Logistic Regression code, second step </h3>
|
||||
In order to get started, we will now replace in our standard ordinary
|
||||
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
|
||||
algorithm with our own SGD code. You can choose whether you want to
|
||||
add the momentum SGD optionality or other SGD variants such as RMSprop
|
||||
or ADAgrad. The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html" target="_self">week 40 contain more
|
||||
details</a>
|
||||
|
||||
<p>
|
||||
Now you should write your Logistic Regression code with the aim to
|
||||
reproduce the Logistic Regression analysis of the <a href="https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf" target="_self">scientific
|
||||
article</a>.
|
||||
Perform an analysis of the results for OLS and Ridge regression as
|
||||
function of the chosen learning rates, the number of mini-batches and
|
||||
epochs as well as algorithm for scaling the learning rate. You can
|
||||
also compare your own results with those that can be obtained using
|
||||
for example <b>Scikit-Learn</b>'s various SGD options. Discuss your
|
||||
results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and
|
||||
the learning rate \( \gamma \). Discuss your results.
|
||||
|
||||
<p>
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
|
||||
|
||||
<h3 id="___sec2" class="anchor">Part b): Writing your own Neural Network code </h3>
|
||||
|
||||
<p>
|
||||
In order to find the optimal parameters of your logistic regressor you
|
||||
should include a gradient descent solver, as discussed in the
|
||||
<a href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html" target="_self">gradient descent
|
||||
lectures</a>.
|
||||
Since we don't have so many data points, you may just code the
|
||||
standard gradient descent with a given learning rate, or even attempt
|
||||
to use the Newton-Raphson method. Alternatively, it may be useful for
|
||||
the next part on neural networks to implement a stochastic gradient
|
||||
descent with and without mini-batches. Stochastic gradient with
|
||||
mini-batches may give the best results. You could finally compare your
|
||||
code with the output from <b>scikit-learn</b>'s toolbox for optimization
|
||||
methods applied to logistic regression.
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the lecture slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_self">week 41</a>.
|
||||
|
||||
<p>
|
||||
We will focus on a regression problem first and study either the
|
||||
Franke function or terrain data (or both or other data sets) from
|
||||
project 1. Discuss again your choice of cost function.
|
||||
|
||||
<p>
|
||||
Write an FFNN code for regression with a flexible number of hidden
|
||||
layers and nodes using the Sigmoid function as activation function for
|
||||
the hidden layers. Initialize the weights using a normal
|
||||
distribution. How would you initialize the biases? And which
|
||||
activation function would you select for the final output layer?
|
||||
|
||||
<p>
|
||||
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
|
||||
You should test your results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes from week 41) or <b>tensorflow/keras</b>.
|
||||
|
||||
<p>
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Linear Regression code and your own Neural Network
|
||||
code. Compare the results with those from project 1.
|
||||
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores.
|
||||
|
||||
<p>
|
||||
A useful reference on the back progagation algorithm is <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's
|
||||
book</a>. It is an excellent
|
||||
read.
|
||||
|
||||
<h3 id="___sec3" class="anchor">Part c): Testing different activation functions </h3>
|
||||
|
||||
<p>
|
||||
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
|
||||
|
||||
<h3 id="___sec4" class="anchor">Part d): Classification analysis using neural networks </h3>
|
||||
|
||||
<p>
|
||||
With a well-written code it should now be easy to change the
|
||||
activation function for the output layer.
|
||||
|
||||
<p>
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in parts b) and c) in order to perform a classification analysis.
|
||||
|
||||
<p>
|
||||
We will here study the MNIST data set of hand-written numbers as
|
||||
discussed in the lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_self">week
|
||||
41</a>. Use
|
||||
the <b>Softmax</b> function as activation function. Your code should
|
||||
however also be able to use a binary activation function as well.
|
||||
|
||||
<p>
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called <em>accuracy</em> score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets \( t_i \) divided by the total
|
||||
number of targets. A perfect classifier will have an accuracy score of
|
||||
\( 1 \).
|
||||
number of targets, that is
|
||||
|
||||
$$
|
||||
\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} ,
|
||||
@@ -239,68 +292,41 @@ $$
|
||||
|
||||
<p>
|
||||
where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \)
|
||||
otherwise if we have a binary classifcation problem. Here \( t_i \)
|
||||
represents the target and \( y_i \) the outputs of your Logistic
|
||||
Regression code.
|
||||
otherwise if we have a binary classification problem. Here \( t_i \)
|
||||
represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).
|
||||
|
||||
<p>
|
||||
You can compare your own results with those obtained using
|
||||
<b>scikit-learn</b>.
|
||||
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
|
||||
|
||||
<p>
|
||||
As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \
|
||||
Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \
|
||||
information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn\
|
||||
.datasets.load_breast_cancer.html" target="_self">Scikit-Learn site</a> or at the "University of California at Irvine":"https://archive.ics.uci.e\
|
||||
du/ml/datasets/breast+cancer+wisconsin+(original)". The <a href="https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\
|
||||
ed.html" target="_self">lecture slides on dimensionality reduction have sev\
|
||||
eral code examples on this data set</a>.
|
||||
|
||||
<h3 id="___sec3" class="anchor">Part c): Writing your own Neural Network code </h3>
|
||||
As stated in the introduction, it can also be useful to study other
|
||||
datasets. In particular, the so-called Wisconsin Cancer
|
||||
data (which is a binary problem, benign or malignant tumors) may be
|
||||
studied. You find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_self">Scikit-Learn
|
||||
site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_self">University of California
|
||||
at Irvine</a>.
|
||||
|
||||
<p>
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write to your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the <a href="https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html" target="_self">lecture
|
||||
slides</a>. We
|
||||
start with the Logistic Regression case and the data set discussed in parts a) and b) but train
|
||||
now the network to find the optimal weights and biases. You are free
|
||||
to use the codes in the above lecture slides as starting points.
|
||||
Again, we strongly recommend that you compare your own neural Network
|
||||
code for classification and pertinent results against a similar code using <b>Scikit-Learn</b> or <b>tensorflow/keras</b> or <b>pytorch</b>.
|
||||
|
||||
<h3 id="___sec5" class="anchor">Part e): Write your Logistic Regression code, final step </h3>
|
||||
|
||||
<p>
|
||||
Discuss again your choice of cost function.
|
||||
Finally, we want to compare the FFNN code we have developed with
|
||||
Logistic regression, that is we wish to compare our neural network
|
||||
classification results with the results we can obtain with another
|
||||
method.
|
||||
|
||||
<p>
|
||||
Train your network and compare the results with those from your Logistic Regression code.
|
||||
You should test your results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes) or <b>tensorflow/keras</b>.
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.
|
||||
Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using <b>Scikit-Learn</b>'s logistic regression functionality.
|
||||
|
||||
<p>
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Logistic Regression code and your own Neural Network
|
||||
code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.
|
||||
The weblink here <a href="https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3" target="_self"><tt>https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3</tt></a>compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
|
||||
|
||||
<p>
|
||||
A useful reference on the back progagation algorithm is <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's
|
||||
book</a>. It is an excellent
|
||||
read.
|
||||
|
||||
<h3 id="___sec4" class="anchor">Part d): Regression analysis using neural networks </h3>
|
||||
|
||||
<p>
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in part c) in order to perform a regression (fitting a
|
||||
function or some data set) analysis. As stated above, our default data
|
||||
sets could be either the Franke function or the terrain data from
|
||||
project 1.
|
||||
|
||||
<p>
|
||||
Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)
|
||||
with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.
|
||||
|
||||
<p>
|
||||
Again, we strongly recommend that you compare your own neural Network code and results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes) or <b>tensorflow/keras</b>.
|
||||
|
||||
<h3 id="___sec5" class="anchor">Part e) Critical evaluation of the various algorithms </h3>
|
||||
<h3 id="___sec6" class="anchor">Part f) Critical evaluation of the various algorithms </h3>
|
||||
|
||||
<p>
|
||||
After all these glorious calculations, you should now summarize the
|
||||
@@ -309,7 +335,7 @@ and cons. Which algorithm works best for the regression case and which
|
||||
is best for the classification case. These codes can also be part of
|
||||
your final project 3, but now applied to other data sets.
|
||||
|
||||
<h2 id="___sec6" class="anchor">Background literature </h2>
|
||||
<h2 id="___sec7" class="anchor">Background literature </h2>
|
||||
|
||||
<ol>
|
||||
<li> The text of Michael Nielsen is highly recommended, see <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's book</a>. It is an excellent read.</li>
|
||||
@@ -317,7 +343,7 @@ your final project 3, but now applied to other data sets.
|
||||
<li> <a href="https://arxiv.org/abs/1803.08823" target="_self">Mehta et al, arXiv 1803.08823</a>, <em>A high-bias, low-variance introduction to Machine Learning for physicists</em>, ArXiv:1803.08823.</li>
|
||||
</ol>
|
||||
|
||||
<h2 id="___sec7" class="anchor">Introduction to numerical projects </h2>
|
||||
<h2 id="___sec8" class="anchor">Introduction to numerical projects </h2>
|
||||
|
||||
<p>
|
||||
Here follows a brief recipe and recommendation on how to write a report for each
|
||||
@@ -335,7 +361,7 @@ project.
|
||||
<li> Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="___sec8" class="anchor">Format for electronic delivery of report and programs </h2>
|
||||
<h2 id="___sec9" class="anchor">Format for electronic delivery of report and programs </h2>
|
||||
|
||||
<p>
|
||||
The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:
|
||||
@@ -350,7 +376,7 @@ Finally,
|
||||
we encourage you to collaborate. Optimal working groups consist of
|
||||
2-3 students. You can then hand in a common report.
|
||||
|
||||
<h2 id="___sec9" class="anchor">Software and needed installations </h2>
|
||||
<h2 id="___sec10" class="anchor">Software and needed installations </h2>
|
||||
|
||||
<p>
|
||||
If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages,
|
||||
|
||||
@@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Project 2 on Machine Learning, deadline November 7">
|
||||
<meta name="description" content="Project 2 on Machine Learning, deadline November 9">
|
||||
|
||||
<title>Project 2 on Machine Learning, deadline November 7</title>
|
||||
<title>Project 2 on Machine Learning, deadline November 9</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
@@ -46,33 +46,38 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Part a): Write your Logistic Regression code, first step',
|
||||
('Part a): Write your own Stochastic Gradient Descent code, '
|
||||
'first step',
|
||||
3,
|
||||
None,
|
||||
'___sec1'),
|
||||
('Part b): Write your Logistic Regression code, second step',
|
||||
('Part b): Writing your own Neural Network code',
|
||||
3,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Part c): Writing your own Neural Network code',
|
||||
('Part c): Testing different activation functions',
|
||||
3,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Part d): Regression analysis using neural networks',
|
||||
('Part d): Classification analysis using neural networks',
|
||||
3,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Part e) Critical evaluation of the various algorithms',
|
||||
('Part e): Write your Logistic Regression code, final step',
|
||||
3,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Background literature', 2, None, '___sec6'),
|
||||
('Introduction to numerical projects', 2, None, '___sec7'),
|
||||
('Part f) Critical evaluation of the various algorithms',
|
||||
3,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Background literature', 2, None, '___sec7'),
|
||||
('Introduction to numerical projects', 2, None, '___sec8'),
|
||||
('Format for electronic delivery of report and programs',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Software and needed installations', 2, None, '___sec9')]}
|
||||
'___sec9'),
|
||||
('Software and needed installations', 2, None, '___sec10')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -102,7 +107,7 @@ MathJax.Hub.Config({
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Project2-bs.html">Project 2 on Machine Learning, deadline November 7</a>
|
||||
<a class="navbar-brand" href="Project2-bs.html">Project 2 on Machine Learning, deadline November 9</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
@@ -111,15 +116,16 @@ MathJax.Hub.Config({
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;"><b>Classification and Regression, from linear and logistic regression to neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;"> Part a): Write your Logistic Regression code, first step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;"> Part b): Write your Logistic Regression code, second step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;"> Part c): Writing your own Neural Network code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;"> Part d): Regression analysis using neural networks</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;"> Part e) Critical evaluation of the various algorithms</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;"><b>Background literature</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;"><b>Introduction to numerical projects</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;"><b>Format for electronic delivery of report and programs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;"><b>Software and needed installations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;"> Part a): Write your own Stochastic Gradient Descent code, first step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;"> Part b): Writing your own Neural Network code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;"> Part c): Testing different activation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;"> Part d): Classification analysis using neural networks</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;"> Part e): Write your Logistic Regression code, final step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;"> Part f) Critical evaluation of the various algorithms</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;"><b>Background literature</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;"><b>Introduction to numerical projects</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;"><b>Format for electronic delivery of report and programs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;"><b>Software and needed installations</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -138,7 +144,7 @@ MathJax.Hub.Config({
|
||||
|
||||
|
||||
<div class="jumbotron">
|
||||
<center><h1>Project 2 on Machine Learning, deadline November 7</h1></center> <!-- document title -->
|
||||
<center><h1>Project 2 on Machine Learning, deadline November 9</h1></center> <!-- document title -->
|
||||
|
||||
<p>
|
||||
<!-- author(s): <a href="http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" target="_self">Data Analysis and Machine Learning FYS-STK3155/FYS4155</a> -->
|
||||
@@ -153,7 +159,7 @@ MathJax.Hub.Config({
|
||||
<center><b>Department of Physics, University of Oslo, Norway</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 6, 2020</h4></center> <!-- date -->
|
||||
<center><h4>Oct 7, 2020</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
</div> <!-- end jumbotron -->
|
||||
@@ -163,8 +169,8 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
The main aim of this project is to study both classification and
|
||||
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
|
||||
in project 1. We will include logistic regresion for classification
|
||||
problems and write our own FFNNcode for studying
|
||||
in project 1. We will also include logistic regression for classification
|
||||
problems and write our own FFNN code for studying
|
||||
both regression and classification problems. The codes developed in
|
||||
project 1, including bootstrap and/or cross-validation as well as the
|
||||
computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can
|
||||
@@ -174,64 +180,111 @@ also be utilized in the present analysis.
|
||||
The data sets that we propose here are (the default sets)
|
||||
|
||||
<ul>
|
||||
<li> Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be
|
||||
<li> Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
|
||||
|
||||
<ol type="a"></li>
|
||||
<li> Either the Franke function or the terrain data from project 1, or data sets your propose.</li>
|
||||
</ol>
|
||||
|
||||
<li> Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine.</li>
|
||||
<li> Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called <a href="https://en.wikipedia.org/wiki/MNIST_database" target="_self">MNIST</a> data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_self">week 41</a></li>
|
||||
</ul>
|
||||
|
||||
However, if you would like to study other data sets, feel free to
|
||||
propose other sets. What we listed here are mere suggestions from our
|
||||
side. If you opt for another data set, consider using a set which
|
||||
has been studied in the scientific literature. This makes it easier
|
||||
for you to compare and analyze your results. It is also an essential
|
||||
elements of the scientific discussion.
|
||||
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
|
||||
element of the scientific discussion.
|
||||
|
||||
<p>
|
||||
In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_self">Scikit-Learn site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_self">University of California at Irvine</a>. The <a href="https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html" target="_self">lecture slides on dimensionality reduction have several code examples on this data set</a>.
|
||||
|
||||
<h3 id="___sec1" class="anchor">Part a): Write your Logistic Regression code, first step </h3>
|
||||
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_self">Scikit-Learn site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_self">University of California at Irvine</a>.
|
||||
|
||||
<p>
|
||||
If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article.
|
||||
We recommend also that you study the code example in the <a href="https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html" target="_self">Logistic Regression</a>.
|
||||
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
|
||||
|
||||
<h3 id="___sec1" class="anchor">Part a): Write your own Stochastic Gradient Descent code, first step </h3>
|
||||
|
||||
<p>
|
||||
Write the part of the code which reads in the data and sets up the relevant data sets.
|
||||
|
||||
<h3 id="___sec2" class="anchor">Part b): Write your Logistic Regression code, second step </h3>
|
||||
In order to get started, we will now replace in our standard ordinary
|
||||
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
|
||||
algorithm with our own SGD code. You can choose whether you want to
|
||||
add the momentum SGD optionality or other SGD variants such as RMSprop
|
||||
or ADAgrad. The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html" target="_self">week 40 contain more
|
||||
details</a>
|
||||
|
||||
<p>
|
||||
Now you should write your Logistic Regression code with the aim to
|
||||
reproduce the Logistic Regression analysis of the <a href="https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf" target="_self">scientific
|
||||
article</a>.
|
||||
Perform an analysis of the results for OLS and Ridge regression as
|
||||
function of the chosen learning rates, the number of mini-batches and
|
||||
epochs as well as algorithm for scaling the learning rate. You can
|
||||
also compare your own results with those that can be obtained using
|
||||
for example <b>Scikit-Learn</b>'s various SGD options. Discuss your
|
||||
results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and
|
||||
the learning rate \( \gamma \). Discuss your results.
|
||||
|
||||
<p>
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
|
||||
|
||||
<h3 id="___sec2" class="anchor">Part b): Writing your own Neural Network code </h3>
|
||||
|
||||
<p>
|
||||
In order to find the optimal parameters of your logistic regressor you
|
||||
should include a gradient descent solver, as discussed in the
|
||||
<a href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html" target="_self">gradient descent
|
||||
lectures</a>.
|
||||
Since we don't have so many data points, you may just code the
|
||||
standard gradient descent with a given learning rate, or even attempt
|
||||
to use the Newton-Raphson method. Alternatively, it may be useful for
|
||||
the next part on neural networks to implement a stochastic gradient
|
||||
descent with and without mini-batches. Stochastic gradient with
|
||||
mini-batches may give the best results. You could finally compare your
|
||||
code with the output from <b>scikit-learn</b>'s toolbox for optimization
|
||||
methods applied to logistic regression.
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the lecture slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_self">week 41</a>.
|
||||
|
||||
<p>
|
||||
We will focus on a regression problem first and study either the
|
||||
Franke function or terrain data (or both or other data sets) from
|
||||
project 1. Discuss again your choice of cost function.
|
||||
|
||||
<p>
|
||||
Write an FFNN code for regression with a flexible number of hidden
|
||||
layers and nodes using the Sigmoid function as activation function for
|
||||
the hidden layers. Initialize the weights using a normal
|
||||
distribution. How would you initialize the biases? And which
|
||||
activation function would you select for the final output layer?
|
||||
|
||||
<p>
|
||||
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
|
||||
You should test your results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes from week 41) or <b>tensorflow/keras</b>.
|
||||
|
||||
<p>
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Linear Regression code and your own Neural Network
|
||||
code. Compare the results with those from project 1.
|
||||
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores.
|
||||
|
||||
<p>
|
||||
A useful reference on the back progagation algorithm is <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's
|
||||
book</a>. It is an excellent
|
||||
read.
|
||||
|
||||
<h3 id="___sec3" class="anchor">Part c): Testing different activation functions </h3>
|
||||
|
||||
<p>
|
||||
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
|
||||
|
||||
<h3 id="___sec4" class="anchor">Part d): Classification analysis using neural networks </h3>
|
||||
|
||||
<p>
|
||||
With a well-written code it should now be easy to change the
|
||||
activation function for the output layer.
|
||||
|
||||
<p>
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in parts b) and c) in order to perform a classification analysis.
|
||||
|
||||
<p>
|
||||
We will here study the MNIST data set of hand-written numbers as
|
||||
discussed in the lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_self">week
|
||||
41</a>. Use
|
||||
the <b>Softmax</b> function as activation function. Your code should
|
||||
however also be able to use a binary activation function as well.
|
||||
|
||||
<p>
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called <em>accuracy</em> score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets \( t_i \) divided by the total
|
||||
number of targets. A perfect classifier will have an accuracy score of
|
||||
\( 1 \).
|
||||
number of targets, that is
|
||||
|
||||
$$
|
||||
\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} ,
|
||||
@@ -239,68 +292,41 @@ $$
|
||||
|
||||
<p>
|
||||
where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \)
|
||||
otherwise if we have a binary classifcation problem. Here \( t_i \)
|
||||
represents the target and \( y_i \) the outputs of your Logistic
|
||||
Regression code.
|
||||
otherwise if we have a binary classification problem. Here \( t_i \)
|
||||
represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).
|
||||
|
||||
<p>
|
||||
You can compare your own results with those obtained using
|
||||
<b>scikit-learn</b>.
|
||||
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
|
||||
|
||||
<p>
|
||||
As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \
|
||||
Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \
|
||||
information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn\
|
||||
.datasets.load_breast_cancer.html" target="_self">Scikit-Learn site</a> or at the "University of California at Irvine":"https://archive.ics.uci.e\
|
||||
du/ml/datasets/breast+cancer+wisconsin+(original)". The <a href="https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\
|
||||
ed.html" target="_self">lecture slides on dimensionality reduction have sev\
|
||||
eral code examples on this data set</a>.
|
||||
|
||||
<h3 id="___sec3" class="anchor">Part c): Writing your own Neural Network code </h3>
|
||||
As stated in the introduction, it can also be useful to study other
|
||||
datasets. In particular, the so-called Wisconsin Cancer
|
||||
data (which is a binary problem, benign or malignant tumors) may be
|
||||
studied. You find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_self">Scikit-Learn
|
||||
site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_self">University of California
|
||||
at Irvine</a>.
|
||||
|
||||
<p>
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write to your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the <a href="https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html" target="_self">lecture
|
||||
slides</a>. We
|
||||
start with the Logistic Regression case and the data set discussed in parts a) and b) but train
|
||||
now the network to find the optimal weights and biases. You are free
|
||||
to use the codes in the above lecture slides as starting points.
|
||||
Again, we strongly recommend that you compare your own neural Network
|
||||
code for classification and pertinent results against a similar code using <b>Scikit-Learn</b> or <b>tensorflow/keras</b> or <b>pytorch</b>.
|
||||
|
||||
<h3 id="___sec5" class="anchor">Part e): Write your Logistic Regression code, final step </h3>
|
||||
|
||||
<p>
|
||||
Discuss again your choice of cost function.
|
||||
Finally, we want to compare the FFNN code we have developed with
|
||||
Logistic regression, that is we wish to compare our neural network
|
||||
classification results with the results we can obtain with another
|
||||
method.
|
||||
|
||||
<p>
|
||||
Train your network and compare the results with those from your Logistic Regression code.
|
||||
You should test your results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes) or <b>tensorflow/keras</b>.
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.
|
||||
Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using <b>Scikit-Learn</b>'s logistic regression functionality.
|
||||
|
||||
<p>
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Logistic Regression code and your own Neural Network
|
||||
code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.
|
||||
The weblink here <a href="https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3" target="_self"><tt>https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3</tt></a>compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
|
||||
|
||||
<p>
|
||||
A useful reference on the back progagation algorithm is <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's
|
||||
book</a>. It is an excellent
|
||||
read.
|
||||
|
||||
<h3 id="___sec4" class="anchor">Part d): Regression analysis using neural networks </h3>
|
||||
|
||||
<p>
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in part c) in order to perform a regression (fitting a
|
||||
function or some data set) analysis. As stated above, our default data
|
||||
sets could be either the Franke function or the terrain data from
|
||||
project 1.
|
||||
|
||||
<p>
|
||||
Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)
|
||||
with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.
|
||||
|
||||
<p>
|
||||
Again, we strongly recommend that you compare your own neural Network code and results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes) or <b>tensorflow/keras</b>.
|
||||
|
||||
<h3 id="___sec5" class="anchor">Part e) Critical evaluation of the various algorithms </h3>
|
||||
<h3 id="___sec6" class="anchor">Part f) Critical evaluation of the various algorithms </h3>
|
||||
|
||||
<p>
|
||||
After all these glorious calculations, you should now summarize the
|
||||
@@ -309,7 +335,7 @@ and cons. Which algorithm works best for the regression case and which
|
||||
is best for the classification case. These codes can also be part of
|
||||
your final project 3, but now applied to other data sets.
|
||||
|
||||
<h2 id="___sec6" class="anchor">Background literature </h2>
|
||||
<h2 id="___sec7" class="anchor">Background literature </h2>
|
||||
|
||||
<ol>
|
||||
<li> The text of Michael Nielsen is highly recommended, see <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's book</a>. It is an excellent read.</li>
|
||||
@@ -317,7 +343,7 @@ your final project 3, but now applied to other data sets.
|
||||
<li> <a href="https://arxiv.org/abs/1803.08823" target="_self">Mehta et al, arXiv 1803.08823</a>, <em>A high-bias, low-variance introduction to Machine Learning for physicists</em>, ArXiv:1803.08823.</li>
|
||||
</ol>
|
||||
|
||||
<h2 id="___sec7" class="anchor">Introduction to numerical projects </h2>
|
||||
<h2 id="___sec8" class="anchor">Introduction to numerical projects </h2>
|
||||
|
||||
<p>
|
||||
Here follows a brief recipe and recommendation on how to write a report for each
|
||||
@@ -335,7 +361,7 @@ project.
|
||||
<li> Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="___sec8" class="anchor">Format for electronic delivery of report and programs </h2>
|
||||
<h2 id="___sec9" class="anchor">Format for electronic delivery of report and programs </h2>
|
||||
|
||||
<p>
|
||||
The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:
|
||||
@@ -350,7 +376,7 @@ Finally,
|
||||
we encourage you to collaborate. Optimal working groups consist of
|
||||
2-3 students. You can then hand in a common report.
|
||||
|
||||
<h2 id="___sec9" class="anchor">Software and needed installations </h2>
|
||||
<h2 id="___sec10" class="anchor">Software and needed installations </h2>
|
||||
|
||||
<p>
|
||||
If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages,
|
||||
|
||||
@@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Project 2 on Machine Learning, deadline November 7">
|
||||
<meta name="description" content="Project 2 on Machine Learning, deadline November 9">
|
||||
|
||||
<title>Project 2 on Machine Learning, deadline November 7</title>
|
||||
<title>Project 2 on Machine Learning, deadline November 9</title>
|
||||
|
||||
|
||||
<style type="text/css">
|
||||
@@ -45,33 +45,38 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Part a): Write your Logistic Regression code, first step',
|
||||
('Part a): Write your own Stochastic Gradient Descent code, '
|
||||
'first step',
|
||||
3,
|
||||
None,
|
||||
'___sec1'),
|
||||
('Part b): Write your Logistic Regression code, second step',
|
||||
('Part b): Writing your own Neural Network code',
|
||||
3,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Part c): Writing your own Neural Network code',
|
||||
('Part c): Testing different activation functions',
|
||||
3,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Part d): Regression analysis using neural networks',
|
||||
('Part d): Classification analysis using neural networks',
|
||||
3,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Part e) Critical evaluation of the various algorithms',
|
||||
('Part e): Write your Logistic Regression code, final step',
|
||||
3,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Background literature', 2, None, '___sec6'),
|
||||
('Introduction to numerical projects', 2, None, '___sec7'),
|
||||
('Part f) Critical evaluation of the various algorithms',
|
||||
3,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Background literature', 2, None, '___sec7'),
|
||||
('Introduction to numerical projects', 2, None, '___sec8'),
|
||||
('Format for electronic delivery of report and programs',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Software and needed installations', 2, None, '___sec9')]}
|
||||
'___sec9'),
|
||||
('Software and needed installations', 2, None, '___sec10')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -97,7 +102,7 @@ MathJax.Hub.Config({
|
||||
|
||||
|
||||
|
||||
<center><h1>Project 2 on Machine Learning, deadline November 7</h1></center> <!-- document title -->
|
||||
<center><h1>Project 2 on Machine Learning, deadline November 9</h1></center> <!-- document title -->
|
||||
|
||||
<p>
|
||||
<!-- author(s): <a href="http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" target="_blank">Data Analysis and Machine Learning FYS-STK3155/FYS4155</a> -->
|
||||
@@ -112,7 +117,7 @@ MathJax.Hub.Config({
|
||||
<center><b>Department of Physics, University of Oslo, Norway</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 6, 2020</h4></center> <!-- date -->
|
||||
<center><h4>Oct 7, 2020</h4></center> <!-- date -->
|
||||
<br>
|
||||
|
||||
<h2 id="___sec0">Classification and Regression, from linear and logistic regression to neural networks </h2>
|
||||
@@ -120,8 +125,8 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
The main aim of this project is to study both classification and
|
||||
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
|
||||
in project 1. We will include logistic regresion for classification
|
||||
problems and write our own FFNNcode for studying
|
||||
in project 1. We will also include logistic regression for classification
|
||||
problems and write our own FFNN code for studying
|
||||
both regression and classification problems. The codes developed in
|
||||
project 1, including bootstrap and/or cross-validation as well as the
|
||||
computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can
|
||||
@@ -131,64 +136,111 @@ also be utilized in the present analysis.
|
||||
The data sets that we propose here are (the default sets)
|
||||
|
||||
<ul>
|
||||
<li> Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be
|
||||
<li> Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
|
||||
|
||||
<ol type="a"></li>
|
||||
<li> Either the Franke function or the terrain data from project 1, or data sets your propose.</li>
|
||||
</ol>
|
||||
|
||||
<li> Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine.</li>
|
||||
<li> Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called <a href="https://en.wikipedia.org/wiki/MNIST_database" target="_blank">MNIST</a> data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_blank">week 41</a></li>
|
||||
</ul>
|
||||
|
||||
However, if you would like to study other data sets, feel free to
|
||||
propose other sets. What we listed here are mere suggestions from our
|
||||
side. If you opt for another data set, consider using a set which
|
||||
has been studied in the scientific literature. This makes it easier
|
||||
for you to compare and analyze your results. It is also an essential
|
||||
elements of the scientific discussion.
|
||||
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
|
||||
element of the scientific discussion.
|
||||
|
||||
<p>
|
||||
In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_blank">Scikit-Learn site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_blank">University of California at Irvine</a>. The <a href="https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html" target="_blank">lecture slides on dimensionality reduction have several code examples on this data set</a>.
|
||||
|
||||
<h3 id="___sec1">Part a): Write your Logistic Regression code, first step </h3>
|
||||
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_blank">Scikit-Learn site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_blank">University of California at Irvine</a>.
|
||||
|
||||
<p>
|
||||
If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article.
|
||||
We recommend also that you study the code example in the <a href="https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html" target="_blank">Logistic Regression</a>.
|
||||
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
|
||||
|
||||
<h3 id="___sec1">Part a): Write your own Stochastic Gradient Descent code, first step </h3>
|
||||
|
||||
<p>
|
||||
Write the part of the code which reads in the data and sets up the relevant data sets.
|
||||
|
||||
<h3 id="___sec2">Part b): Write your Logistic Regression code, second step </h3>
|
||||
In order to get started, we will now replace in our standard ordinary
|
||||
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
|
||||
algorithm with our own SGD code. You can choose whether you want to
|
||||
add the momentum SGD optionality or other SGD variants such as RMSprop
|
||||
or ADAgrad. The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html" target="_blank">week 40 contain more
|
||||
details</a>
|
||||
|
||||
<p>
|
||||
Now you should write your Logistic Regression code with the aim to
|
||||
reproduce the Logistic Regression analysis of the <a href="https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf" target="_blank">scientific
|
||||
article</a>.
|
||||
Perform an analysis of the results for OLS and Ridge regression as
|
||||
function of the chosen learning rates, the number of mini-batches and
|
||||
epochs as well as algorithm for scaling the learning rate. You can
|
||||
also compare your own results with those that can be obtained using
|
||||
for example <b>Scikit-Learn</b>'s various SGD options. Discuss your
|
||||
results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and
|
||||
the learning rate \( \gamma \). Discuss your results.
|
||||
|
||||
<p>
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
|
||||
|
||||
<h3 id="___sec2">Part b): Writing your own Neural Network code </h3>
|
||||
|
||||
<p>
|
||||
In order to find the optimal parameters of your logistic regressor you
|
||||
should include a gradient descent solver, as discussed in the
|
||||
<a href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html" target="_blank">gradient descent
|
||||
lectures</a>.
|
||||
Since we don't have so many data points, you may just code the
|
||||
standard gradient descent with a given learning rate, or even attempt
|
||||
to use the Newton-Raphson method. Alternatively, it may be useful for
|
||||
the next part on neural networks to implement a stochastic gradient
|
||||
descent with and without mini-batches. Stochastic gradient with
|
||||
mini-batches may give the best results. You could finally compare your
|
||||
code with the output from <b>scikit-learn</b>'s toolbox for optimization
|
||||
methods applied to logistic regression.
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the lecture slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_blank">week 41</a>.
|
||||
|
||||
<p>
|
||||
We will focus on a regression problem first and study either the
|
||||
Franke function or terrain data (or both or other data sets) from
|
||||
project 1. Discuss again your choice of cost function.
|
||||
|
||||
<p>
|
||||
Write an FFNN code for regression with a flexible number of hidden
|
||||
layers and nodes using the Sigmoid function as activation function for
|
||||
the hidden layers. Initialize the weights using a normal
|
||||
distribution. How would you initialize the biases? And which
|
||||
activation function would you select for the final output layer?
|
||||
|
||||
<p>
|
||||
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
|
||||
You should test your results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes from week 41) or <b>tensorflow/keras</b>.
|
||||
|
||||
<p>
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Linear Regression code and your own Neural Network
|
||||
code. Compare the results with those from project 1.
|
||||
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores.
|
||||
|
||||
<p>
|
||||
A useful reference on the back progagation algorithm is <a href="http://neuralnetworksanddeeplearning.com/" target="_blank">Nielsen's
|
||||
book</a>. It is an excellent
|
||||
read.
|
||||
|
||||
<h3 id="___sec3">Part c): Testing different activation functions </h3>
|
||||
|
||||
<p>
|
||||
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
|
||||
|
||||
<h3 id="___sec4">Part d): Classification analysis using neural networks </h3>
|
||||
|
||||
<p>
|
||||
With a well-written code it should now be easy to change the
|
||||
activation function for the output layer.
|
||||
|
||||
<p>
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in parts b) and c) in order to perform a classification analysis.
|
||||
|
||||
<p>
|
||||
We will here study the MNIST data set of hand-written numbers as
|
||||
discussed in the lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html" target="_blank">week
|
||||
41</a>. Use
|
||||
the <b>Softmax</b> function as activation function. Your code should
|
||||
however also be able to use a binary activation function as well.
|
||||
|
||||
<p>
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called <em>accuracy</em> score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets \( t_i \) divided by the total
|
||||
number of targets. A perfect classifier will have an accuracy score of
|
||||
\( 1 \).
|
||||
number of targets, that is
|
||||
|
||||
$$
|
||||
\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} ,
|
||||
@@ -196,68 +248,41 @@ $$
|
||||
|
||||
<p>
|
||||
where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \)
|
||||
otherwise if we have a binary classifcation problem. Here \( t_i \)
|
||||
represents the target and \( y_i \) the outputs of your Logistic
|
||||
Regression code.
|
||||
otherwise if we have a binary classification problem. Here \( t_i \)
|
||||
represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).
|
||||
|
||||
<p>
|
||||
You can compare your own results with those obtained using
|
||||
<b>scikit-learn</b>.
|
||||
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
|
||||
|
||||
<p>
|
||||
As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \
|
||||
Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \
|
||||
information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn\
|
||||
.datasets.load_breast_cancer.html" target="_blank">Scikit-Learn site</a> or at the "University of California at Irvine":"https://archive.ics.uci.e\
|
||||
du/ml/datasets/breast+cancer+wisconsin+(original)". The <a href="https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\
|
||||
ed.html" target="_blank">lecture slides on dimensionality reduction have sev\
|
||||
eral code examples on this data set</a>.
|
||||
|
||||
<h3 id="___sec3">Part c): Writing your own Neural Network code </h3>
|
||||
As stated in the introduction, it can also be useful to study other
|
||||
datasets. In particular, the so-called Wisconsin Cancer
|
||||
data (which is a binary problem, benign or malignant tumors) may be
|
||||
studied. You find more information about this at the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" target="_blank">Scikit-Learn
|
||||
site</a> or at the <a href="https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)" target="_blank">University of California
|
||||
at Irvine</a>.
|
||||
|
||||
<p>
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write to your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the <a href="https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html" target="_blank">lecture
|
||||
slides</a>. We
|
||||
start with the Logistic Regression case and the data set discussed in parts a) and b) but train
|
||||
now the network to find the optimal weights and biases. You are free
|
||||
to use the codes in the above lecture slides as starting points.
|
||||
Again, we strongly recommend that you compare your own neural Network
|
||||
code for classification and pertinent results against a similar code using <b>Scikit-Learn</b> or <b>tensorflow/keras</b> or <b>pytorch</b>.
|
||||
|
||||
<h3 id="___sec5">Part e): Write your Logistic Regression code, final step </h3>
|
||||
|
||||
<p>
|
||||
Discuss again your choice of cost function.
|
||||
Finally, we want to compare the FFNN code we have developed with
|
||||
Logistic regression, that is we wish to compare our neural network
|
||||
classification results with the results we can obtain with another
|
||||
method.
|
||||
|
||||
<p>
|
||||
Train your network and compare the results with those from your Logistic Regression code.
|
||||
You should test your results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes) or <b>tensorflow/keras</b>.
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.
|
||||
Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using <b>Scikit-Learn</b>'s logistic regression functionality.
|
||||
|
||||
<p>
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Logistic Regression code and your own Neural Network
|
||||
code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.
|
||||
The weblink here <a href="https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3" target="_blank"><tt>https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3</tt></a>compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
|
||||
|
||||
<p>
|
||||
A useful reference on the back progagation algorithm is <a href="http://neuralnetworksanddeeplearning.com/" target="_blank">Nielsen's
|
||||
book</a>. It is an excellent
|
||||
read.
|
||||
|
||||
<h3 id="___sec4">Part d): Regression analysis using neural networks </h3>
|
||||
|
||||
<p>
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in part c) in order to perform a regression (fitting a
|
||||
function or some data set) analysis. As stated above, our default data
|
||||
sets could be either the Franke function or the terrain data from
|
||||
project 1.
|
||||
|
||||
<p>
|
||||
Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)
|
||||
with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.
|
||||
|
||||
<p>
|
||||
Again, we strongly recommend that you compare your own neural Network code and results against a similar code using <b>Scikit-Learn</b> (see the examples in the above lecture notes) or <b>tensorflow/keras</b>.
|
||||
|
||||
<h3 id="___sec5">Part e) Critical evaluation of the various algorithms </h3>
|
||||
<h3 id="___sec6">Part f) Critical evaluation of the various algorithms </h3>
|
||||
|
||||
<p>
|
||||
After all these glorious calculations, you should now summarize the
|
||||
@@ -266,7 +291,7 @@ and cons. Which algorithm works best for the regression case and which
|
||||
is best for the classification case. These codes can also be part of
|
||||
your final project 3, but now applied to other data sets.
|
||||
|
||||
<h2 id="___sec6">Background literature </h2>
|
||||
<h2 id="___sec7">Background literature </h2>
|
||||
|
||||
<ol>
|
||||
<li> The text of Michael Nielsen is highly recommended, see <a href="http://neuralnetworksanddeeplearning.com/" target="_blank">Nielsen's book</a>. It is an excellent read.</li>
|
||||
@@ -274,7 +299,7 @@ your final project 3, but now applied to other data sets.
|
||||
<li> <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et al, arXiv 1803.08823</a>, <em>A high-bias, low-variance introduction to Machine Learning for physicists</em>, ArXiv:1803.08823.</li>
|
||||
</ol>
|
||||
|
||||
<h2 id="___sec7">Introduction to numerical projects </h2>
|
||||
<h2 id="___sec8">Introduction to numerical projects </h2>
|
||||
|
||||
<p>
|
||||
Here follows a brief recipe and recommendation on how to write a report for each
|
||||
@@ -292,7 +317,7 @@ project.
|
||||
<li> Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="___sec8">Format for electronic delivery of report and programs </h2>
|
||||
<h2 id="___sec9">Format for electronic delivery of report and programs </h2>
|
||||
|
||||
<p>
|
||||
The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:
|
||||
@@ -307,7 +332,7 @@ Finally,
|
||||
we encourage you to collaborate. Optimal working groups consist of
|
||||
2-3 students. You can then hand in a common report.
|
||||
|
||||
<h2 id="___sec9">Software and needed installations </h2>
|
||||
<h2 id="___sec10">Software and needed installations </h2>
|
||||
|
||||
<p>
|
||||
If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages,
|
||||
|
||||
@@ -4,13 +4,13 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<!-- dom:TITLE: Project 2 on Machine Learning, deadline November 7 -->\n",
|
||||
"# Project 2 on Machine Learning, deadline November 7\n",
|
||||
"<!-- dom:TITLE: Project 2 on Machine Learning, deadline November 9 -->\n",
|
||||
"# Project 2 on Machine Learning, deadline November 9\n",
|
||||
"<!-- dom:AUTHOR: [Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html) at Department of Physics, University of Oslo, Norway -->\n",
|
||||
"<!-- Author: --> \n",
|
||||
"**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, Department of Physics, University of Oslo, Norway\n",
|
||||
"\n",
|
||||
"Date: **Oct 6, 2020**\n",
|
||||
"Date: **Oct 7, 2020**\n",
|
||||
"\n",
|
||||
"Copyright 1999-2020, [Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html). Released under CC Attribution-NonCommercial 4.0 license\n",
|
||||
"\n",
|
||||
@@ -21,8 +21,8 @@
|
||||
"\n",
|
||||
"The main aim of this project is to study both classification and\n",
|
||||
"regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied\n",
|
||||
"in project 1. We will include logistic regresion for classification\n",
|
||||
"problems and write our own FFNNcode for studying\n",
|
||||
"in project 1. We will also include logistic regression for classification\n",
|
||||
"problems and write our own FFNN code for studying\n",
|
||||
"both regression and classification problems. The codes developed in\n",
|
||||
"project 1, including bootstrap and/or cross-validation as well as the\n",
|
||||
"computation of the mean-squared error and/or the $R2$ or the accuracy score (classification problems) functions can\n",
|
||||
@@ -31,56 +31,95 @@
|
||||
"\n",
|
||||
"The data sets that we propose here are (the default sets)\n",
|
||||
"\n",
|
||||
"* Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be\n",
|
||||
"* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be\n",
|
||||
"\n",
|
||||
"a. Either the Franke function or the terrain data from project 1, or data sets your propose.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine. \n",
|
||||
"* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called [MNIST](https://en.wikipedia.org/wiki/MNIST_database) data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from [week 41](https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html)\n",
|
||||
"\n",
|
||||
"However, if you would like to study other data sets, feel free to\n",
|
||||
"propose other sets. What we listed here are mere suggestions from our\n",
|
||||
"side. If you opt for another data set, consider using a set which\n",
|
||||
"has been studied in the scientific literature. This makes it easier\n",
|
||||
"for you to compare and analyze your results. It is also an essential\n",
|
||||
"elements of the scientific discussion.\n",
|
||||
"for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential\n",
|
||||
"element of the scientific discussion.\n",
|
||||
"\n",
|
||||
"In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the [Scikit-Learn site](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) or at the [University of California at Irvine](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)). The [lecture slides on dimensionality reduction have several code examples on this data set](https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html).\n",
|
||||
"In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the [Scikit-Learn site](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) or at the [University of California at Irvine](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)). \n",
|
||||
"\n",
|
||||
"### Part a): Write your Logistic Regression code, first step\n",
|
||||
"We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code. \n",
|
||||
"\n",
|
||||
"If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article. \n",
|
||||
"We recommend also that you study the code example in the [Logistic Regression](https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html). \n",
|
||||
"### Part a): Write your own Stochastic Gradient Descent code, first step\n",
|
||||
"\n",
|
||||
"Write the part of the code which reads in the data and sets up the relevant data sets. \n",
|
||||
"In order to get started, we will now replace in our standard ordinary\n",
|
||||
"least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion\n",
|
||||
"algorithm with our own SGD code. You can choose whether you want to\n",
|
||||
"add the momentum SGD optionality or other SGD variants such as RMSprop\n",
|
||||
"or ADAgrad. The lecture notes from [week 40 contain more\n",
|
||||
"details](https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html)\n",
|
||||
"\n",
|
||||
"### Part b): Write your Logistic Regression code, second step\n",
|
||||
"Perform an analysis of the results for OLS and Ridge regression as\n",
|
||||
"function of the chosen learning rates, the number of mini-batches and\n",
|
||||
"epochs as well as algorithm for scaling the learning rate. You can\n",
|
||||
"also compare your own results with those that can be obtained using\n",
|
||||
"for example **Scikit-Learn**'s various SGD options. Discuss your\n",
|
||||
"results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\\lambda$ and \n",
|
||||
"the learning rate $\\gamma$. Discuss your results.\n",
|
||||
"\n",
|
||||
"Now you should write your Logistic Regression code with the aim to\n",
|
||||
"reproduce the Logistic Regression analysis of the [scientific\n",
|
||||
"article](https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf).\n",
|
||||
"You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.\n",
|
||||
"\n",
|
||||
"Define your cost function and the design matrix before you start writing your code.\n",
|
||||
"### Part b): Writing your own Neural Network code\n",
|
||||
"\n",
|
||||
"In order to find the optimal parameters of your logistic regressor you\n",
|
||||
"should include a gradient descent solver, as discussed in the\n",
|
||||
"[gradient descent\n",
|
||||
"lectures](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html).\n",
|
||||
"Since we don't have so many data points, you may just code the\n",
|
||||
"standard gradient descent with a given learning rate, or even attempt\n",
|
||||
"to use the Newton-Raphson method. Alternatively, it may be useful for\n",
|
||||
"the next part on neural networks to implement a stochastic gradient\n",
|
||||
"descent with and without mini-batches. Stochastic gradient with\n",
|
||||
"mini-batches may give the best results. You could finally compare your\n",
|
||||
"code with the output from **scikit-learn**'s toolbox for optimization\n",
|
||||
"methods applied to logistic regression.\n",
|
||||
"Your aim now, and this is the central part of this project, is to\n",
|
||||
"write your own Feed Forward Neural Network code implementing the back\n",
|
||||
"propagation algorithm discussed in the lecture slides from [week 41](https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html).\n",
|
||||
"\n",
|
||||
"We will focus on a regression problem first and study either the\n",
|
||||
"Franke function or terrain data (or both or other data sets) from\n",
|
||||
"project 1. Discuss again your choice of cost function.\n",
|
||||
"\n",
|
||||
"Write an FFNN code for regression with a flexible number of hidden\n",
|
||||
"layers and nodes using the Sigmoid function as activation function for\n",
|
||||
"the hidden layers. Initialize the weights using a normal\n",
|
||||
"distribution. How would you initialize the biases? And which\n",
|
||||
"activation function would you select for the final output layer?\n",
|
||||
"\n",
|
||||
"Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1. \n",
|
||||
"You should test your results against a similar code using **Scikit-Learn** (see the examples in the above lecture notes from week 41) or **tensorflow/keras**. \n",
|
||||
"\n",
|
||||
"Comment your results and give a critical discussion of the results\n",
|
||||
"obtained with the Linear Regression code and your own Neural Network\n",
|
||||
"code. Compare the results with those from project 1.\n",
|
||||
"Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores.\n",
|
||||
"\n",
|
||||
"A useful reference on the back progagation algorithm is [Nielsen's\n",
|
||||
"book](http://neuralnetworksanddeeplearning.com/). It is an excellent\n",
|
||||
"read.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Part c): Testing different activation functions\n",
|
||||
"\n",
|
||||
"You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.\n",
|
||||
"\n",
|
||||
"### Part d): Classification analysis using neural networks\n",
|
||||
"\n",
|
||||
"With a well-written code it should now be easy to change the\n",
|
||||
"activation function for the output layer.\n",
|
||||
"\n",
|
||||
"Here we will change the cost function for our neural network code\n",
|
||||
"developed in parts b) and c) in order to perform a classification analysis. \n",
|
||||
"\n",
|
||||
"We will here study the MNIST data set of hand-written numbers as\n",
|
||||
"discussed in the lecture notes from [week\n",
|
||||
"41](https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html). Use\n",
|
||||
"the **Softmax** function as activation function. Your code should\n",
|
||||
"however also be able to use a binary activation function as well.\n",
|
||||
"\n",
|
||||
"To measure the performance of our classification problem we use the\n",
|
||||
"so-called *accuracy* score. The accuracy is as you would expect just\n",
|
||||
"the number of correctly guessed targets $t_i$ divided by the total\n",
|
||||
"number of targets. A perfect classifier will have an accuracy score of\n",
|
||||
"$1$."
|
||||
"number of targets, that is"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -97,64 +136,44 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$\n",
|
||||
"otherwise if we have a binary classifcation problem. Here $t_i$\n",
|
||||
"represents the target and $y_i$ the outputs of your Logistic\n",
|
||||
"Regression code.\n",
|
||||
"otherwise if we have a binary classification problem. Here $t_i$\n",
|
||||
"represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.\n",
|
||||
"\n",
|
||||
"Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\\lambda$ (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"You can compare your own results with those obtained using\n",
|
||||
"**scikit-learn**.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own\tLogistic Regression code for classification problems, the so-called \\\n",
|
||||
"Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \\\n",
|
||||
"information about this at the [Scikit-Learn site](https://scikit-learn.org/stable/modules/generated/sklearn\\\n",
|
||||
".datasets.load_breast_cancer.html) or at the \"University\tof California at Irvine\":\"https://archive.ics.uci.e\\\n",
|
||||
"du/ml/datasets/breast+cancer+wisconsin+(original)\". The\t[lecture slides on dimensionality reduction have sev\\\n",
|
||||
"eral code examples on this data set](https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\\\n",
|
||||
"ed.html).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Part c): Writing your own Neural Network code\n",
|
||||
"\n",
|
||||
"Your aim now, and this is the central part of this project, is to\n",
|
||||
"write to your own Feed Forward Neural Network code implementing the back\n",
|
||||
"propagation algorithm discussed in the [lecture\n",
|
||||
"slides](https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html). We\n",
|
||||
"start with the Logistic Regression case and the data set discussed in parts a) and b) but train\n",
|
||||
"now the network to find the optimal weights and biases. You are free\n",
|
||||
"to use the codes in the above lecture slides as starting points.\n",
|
||||
"\n",
|
||||
"Discuss again your choice of cost function.\n",
|
||||
"\n",
|
||||
"Train your network and compare the results with those from your Logistic Regression code. \n",
|
||||
"You should test your results against a similar code using **Scikit-Learn** (see the examples in the above lecture notes) or **tensorflow/keras**. \n",
|
||||
"\n",
|
||||
"Comment your results and give a critical discussion of the results\n",
|
||||
"obtained with the Logistic Regression code and your own Neural Network\n",
|
||||
"code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.\n",
|
||||
"\n",
|
||||
"A useful reference on the back progagation algorithm is [Nielsen's\n",
|
||||
"book](http://neuralnetworksanddeeplearning.com/). It is an excellent\n",
|
||||
"read.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Part d): Regression analysis using neural networks\n",
|
||||
"\n",
|
||||
"Here we will change the cost function for our neural network code\n",
|
||||
"developed in part c) in order to perform a regression (fitting a\n",
|
||||
"function or some data set) analysis. As stated above, our default data\n",
|
||||
"sets could be either the Franke function or the terrain data from\n",
|
||||
"project 1.\n",
|
||||
"\n",
|
||||
"Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)\n",
|
||||
"with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.\n",
|
||||
"\n",
|
||||
"Again, we strongly recommend that you compare your own neural Network code and results against a similar code using **Scikit-Learn** (see the examples in the above lecture notes) or **tensorflow/keras**. \n",
|
||||
"As stated in the introduction, it can also be useful to study other\n",
|
||||
"datasets. In particular, the so-called Wisconsin Cancer\n",
|
||||
"data (which is a binary problem, benign or malignant tumors) may be\n",
|
||||
"studied. You find more information about this at the [Scikit-Learn\n",
|
||||
"site](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) or at the [University of California\n",
|
||||
"at Irvine](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)). \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Part e) Critical evaluation of the various algorithms\n",
|
||||
"\n",
|
||||
"Again, we strongly recommend that you compare your own neural Network\n",
|
||||
"code for classification and pertinent results against a similar code using **Scikit-Learn** or **tensorflow/keras** or **pytorch**.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Part e): Write your Logistic Regression code, final step\n",
|
||||
"\n",
|
||||
"Finally, we want to compare the FFNN code we have developed with\n",
|
||||
"Logistic regression, that is we wish to compare our neural network\n",
|
||||
"classification results with the results we can obtain with another\n",
|
||||
"method.\n",
|
||||
"\n",
|
||||
"Define your cost function and the design matrix before you start writing your code.\n",
|
||||
"Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.\n",
|
||||
"Add also an $l_2$ regularization parameter $\\lambda$. Compare your results with those from your FFNN code as well as those obtained using **Scikit-Learn**'s logistic regression functionality.\n",
|
||||
"\n",
|
||||
"The weblink here <https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3>compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Part f) Critical evaluation of the various algorithms\n",
|
||||
"\n",
|
||||
"After all these glorious calculations, you should now summarize the\n",
|
||||
"various algorithms and come with a critical evaluation of their pros\n",
|
||||
|
||||
Binary file not shown.
@@ -129,7 +129,7 @@ final, % draft: marks overfull hboxes, figures with paths
|
||||
\begin{center}
|
||||
{\LARGE\bf
|
||||
\begin{spacing}{1.25}
|
||||
Project 2 on Machine Learning, deadline November 7
|
||||
Project 2 on Machine Learning, deadline November 9
|
||||
\end{spacing}
|
||||
}
|
||||
\end{center}
|
||||
@@ -149,7 +149,7 @@ Project 2 on Machine Learning, deadline November 7
|
||||
|
||||
% --- begin date ---
|
||||
\begin{center}
|
||||
Oct 6, 2020
|
||||
Oct 7, 2020
|
||||
\end{center}
|
||||
% --- end date ---
|
||||
|
||||
@@ -160,8 +160,8 @@ Oct 6, 2020
|
||||
|
||||
The main aim of this project is to study both classification and
|
||||
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
|
||||
in project 1. We will include logistic regresion for classification
|
||||
problems and write our own FFNNcode for studying
|
||||
in project 1. We will also include logistic regression for classification
|
||||
problems and write our own FFNN code for studying
|
||||
both regression and classification problems. The codes developed in
|
||||
project 1, including bootstrap and/or cross-validation as well as the
|
||||
computation of the mean-squared error and/or the $R2$ or the accuracy score (classification problems) functions can
|
||||
@@ -171,7 +171,7 @@ also be utilized in the present analysis.
|
||||
The data sets that we propose here are (the default sets)
|
||||
|
||||
\begin{itemize}
|
||||
\item Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be
|
||||
\item Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
|
||||
\begin{enumerate}
|
||||
|
||||
\item Either the Franke function or the terrain data from project 1, or data sets your propose.
|
||||
@@ -179,7 +179,7 @@ The data sets that we propose here are (the default sets)
|
||||
\end{enumerate}
|
||||
|
||||
\noindent
|
||||
\item Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine.
|
||||
\item Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called \href{{https://en.wikipedia.org/wiki/MNIST_database}}{MNIST} data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html}}{week 41}
|
||||
\end{itemize}
|
||||
|
||||
\noindent
|
||||
@@ -187,105 +187,124 @@ However, if you would like to study other data sets, feel free to
|
||||
propose other sets. What we listed here are mere suggestions from our
|
||||
side. If you opt for another data set, consider using a set which
|
||||
has been studied in the scientific literature. This makes it easier
|
||||
for you to compare and analyze your results. It is also an essential
|
||||
elements of the scientific discussion.
|
||||
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
|
||||
element of the scientific discussion.
|
||||
|
||||
In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html}}{Scikit-Learn site} or at the \href{{https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)}}{University of California at Irvine}. The \href{{https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html}}{lecture slides on dimensionality reduction have several code examples on this data set}.
|
||||
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html}}{Scikit-Learn site} or at the \href{{https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)}}{University of California at Irvine}.
|
||||
|
||||
\paragraph{Part a): Write your Logistic Regression code, first step.}
|
||||
If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article.
|
||||
We recommend also that you study the code example in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html}}{Logistic Regression}.
|
||||
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
|
||||
|
||||
Write the part of the code which reads in the data and sets up the relevant data sets.
|
||||
\paragraph{Part a): Write your own Stochastic Gradient Descent code, first step.}
|
||||
In order to get started, we will now replace in our standard ordinary
|
||||
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
|
||||
algorithm with our own SGD code. You can choose whether you want to
|
||||
add the momentum SGD optionality or other SGD variants such as RMSprop
|
||||
or ADAgrad. The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html}}{week 40 contain more
|
||||
details}
|
||||
|
||||
\paragraph{Part b): Write your Logistic Regression code, second step.}
|
||||
Now you should write your Logistic Regression code with the aim to
|
||||
reproduce the Logistic Regression analysis of the \href{{https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf}}{scientific
|
||||
article}.
|
||||
Perform an analysis of the results for OLS and Ridge regression as
|
||||
function of the chosen learning rates, the number of mini-batches and
|
||||
epochs as well as algorithm for scaling the learning rate. You can
|
||||
also compare your own results with those that can be obtained using
|
||||
for example \textbf{Scikit-Learn}'s various SGD options. Discuss your
|
||||
results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and
|
||||
the learning rate $\gamma$. Discuss your results.
|
||||
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
|
||||
|
||||
In order to find the optimal parameters of your logistic regressor you
|
||||
should include a gradient descent solver, as discussed in the
|
||||
\href{{https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html}}{gradient descent
|
||||
lectures}.
|
||||
Since we don't have so many data points, you may just code the
|
||||
standard gradient descent with a given learning rate, or even attempt
|
||||
to use the Newton-Raphson method. Alternatively, it may be useful for
|
||||
the next part on neural networks to implement a stochastic gradient
|
||||
descent with and without mini-batches. Stochastic gradient with
|
||||
mini-batches may give the best results. You could finally compare your
|
||||
code with the output from \textbf{scikit-learn}'s toolbox for optimization
|
||||
methods applied to logistic regression.
|
||||
|
||||
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called \emph{accuracy} score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets $t_i$ divided by the total
|
||||
number of targets. A perfect classifier will have an accuracy score of
|
||||
$1$.
|
||||
|
||||
\[
|
||||
\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} ,
|
||||
\]
|
||||
|
||||
where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$
|
||||
otherwise if we have a binary classifcation problem. Here $t_i$
|
||||
represents the target and $y_i$ the outputs of your Logistic
|
||||
Regression code.
|
||||
|
||||
|
||||
You can compare your own results with those obtained using
|
||||
\textbf{scikit-learn}.
|
||||
|
||||
|
||||
As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \
|
||||
Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \
|
||||
information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn\
|
||||
.datasets.load_breast_cancer.html}}{Scikit-Learn site} or at the "University of California at Irvine":"https://archive.ics.uci.e\
|
||||
du/ml/datasets/breast+cancer+wisconsin+(original)". The \href{{https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\
|
||||
ed.html}}{lecture slides on dimensionality reduction have sev\
|
||||
eral code examples on this data set}.
|
||||
|
||||
|
||||
\paragraph{Part c): Writing your own Neural Network code.}
|
||||
\paragraph{Part b): Writing your own Neural Network code.}
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write to your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html}}{lecture
|
||||
slides}. We
|
||||
start with the Logistic Regression case and the data set discussed in parts a) and b) but train
|
||||
now the network to find the optimal weights and biases. You are free
|
||||
to use the codes in the above lecture slides as starting points.
|
||||
write your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the lecture slides from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html}}{week 41}.
|
||||
|
||||
Discuss again your choice of cost function.
|
||||
We will focus on a regression problem first and study either the
|
||||
Franke function or terrain data (or both or other data sets) from
|
||||
project 1. Discuss again your choice of cost function.
|
||||
|
||||
Train your network and compare the results with those from your Logistic Regression code.
|
||||
You should test your results against a similar code using \textbf{Scikit-Learn} (see the examples in the above lecture notes) or \textbf{tensorflow/keras}.
|
||||
Write an FFNN code for regression with a flexible number of hidden
|
||||
layers and nodes using the Sigmoid function as activation function for
|
||||
the hidden layers. Initialize the weights using a normal
|
||||
distribution. How would you initialize the biases? And which
|
||||
activation function would you select for the final output layer?
|
||||
|
||||
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
|
||||
You should test your results against a similar code using \textbf{Scikit-Learn} (see the examples in the above lecture notes from week 41) or \textbf{tensorflow/keras}.
|
||||
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Logistic Regression code and your own Neural Network
|
||||
code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.
|
||||
obtained with the Linear Regression code and your own Neural Network
|
||||
code. Compare the results with those from project 1.
|
||||
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores.
|
||||
|
||||
A useful reference on the back progagation algorithm is \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's
|
||||
book}. It is an excellent
|
||||
read.
|
||||
|
||||
|
||||
\paragraph{Part d): Regression analysis using neural networks.}
|
||||
|
||||
\paragraph{Part c): Testing different activation functions.}
|
||||
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
|
||||
|
||||
\paragraph{Part d): Classification analysis using neural networks.}
|
||||
With a well-written code it should now be easy to change the
|
||||
activation function for the output layer.
|
||||
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in part c) in order to perform a regression (fitting a
|
||||
function or some data set) analysis. As stated above, our default data
|
||||
sets could be either the Franke function or the terrain data from
|
||||
project 1.
|
||||
developed in parts b) and c) in order to perform a classification analysis.
|
||||
|
||||
Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)
|
||||
with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.
|
||||
We will here study the MNIST data set of hand-written numbers as
|
||||
discussed in the lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html}}{week
|
||||
41}. Use
|
||||
the \textbf{Softmax} function as activation function. Your code should
|
||||
however also be able to use a binary activation function as well.
|
||||
|
||||
Again, we strongly recommend that you compare your own neural Network code and results against a similar code using \textbf{Scikit-Learn} (see the examples in the above lecture notes) or \textbf{tensorflow/keras}.
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called \emph{accuracy} score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets $t_i$ divided by the total
|
||||
number of targets, that is
|
||||
|
||||
|
||||
\[
|
||||
\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} ,
|
||||
\]
|
||||
|
||||
where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$
|
||||
otherwise if we have a binary classification problem. Here $t_i$
|
||||
represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.
|
||||
|
||||
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\lambda$ (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
|
||||
|
||||
|
||||
As stated in the introduction, it can also be useful to study other
|
||||
datasets. In particular, the so-called Wisconsin Cancer
|
||||
data (which is a binary problem, benign or malignant tumors) may be
|
||||
studied. You find more information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html}}{Scikit-Learn
|
||||
site} or at the \href{{https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)}}{University of California
|
||||
at Irvine}.
|
||||
|
||||
|
||||
|
||||
\paragraph{Part e) Critical evaluation of the various algorithms.}
|
||||
|
||||
Again, we strongly recommend that you compare your own neural Network
|
||||
code for classification and pertinent results against a similar code using \textbf{Scikit-Learn} or \textbf{tensorflow/keras} or \textbf{pytorch}.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
\paragraph{Part e): Write your Logistic Regression code, final step.}
|
||||
Finally, we want to compare the FFNN code we have developed with
|
||||
Logistic regression, that is we wish to compare our neural network
|
||||
classification results with the results we can obtain with another
|
||||
method.
|
||||
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.
|
||||
Add also an $l_2$ regularization parameter $\lambda$. Compare your results with those from your FFNN code as well as those obtained using \textbf{Scikit-Learn}'s logistic regression functionality.
|
||||
|
||||
The weblink here \href{{https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3}}{\nolinkurl{https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3}}compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
|
||||
|
||||
|
||||
\paragraph{Part f) Critical evaluation of the various algorithms.}
|
||||
After all these glorious calculations, you should now summarize the
|
||||
various algorithms and come with a critical evaluation of their pros
|
||||
and cons. Which algorithm works best for the regression case and which
|
||||
|
||||
Binary file not shown.
@@ -103,7 +103,7 @@ final, % draft: marks overfull hboxes, figures with paths
|
||||
\begin{center}
|
||||
{\LARGE\bf
|
||||
\begin{spacing}{1.25}
|
||||
Project 2 on Machine Learning, deadline November 7
|
||||
Project 2 on Machine Learning, deadline November 9
|
||||
\end{spacing}
|
||||
}
|
||||
\end{center}
|
||||
@@ -123,7 +123,7 @@ Project 2 on Machine Learning, deadline November 7
|
||||
|
||||
% --- begin date ---
|
||||
\begin{center}
|
||||
Oct 6, 2020
|
||||
Oct 7, 2020
|
||||
\end{center}
|
||||
% --- end date ---
|
||||
|
||||
@@ -134,8 +134,8 @@ Oct 6, 2020
|
||||
|
||||
The main aim of this project is to study both classification and
|
||||
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
|
||||
in project 1. We will include logistic regresion for classification
|
||||
problems and write our own FFNNcode for studying
|
||||
in project 1. We will also include logistic regression for classification
|
||||
problems and write our own FFNN code for studying
|
||||
both regression and classification problems. The codes developed in
|
||||
project 1, including bootstrap and/or cross-validation as well as the
|
||||
computation of the mean-squared error and/or the $R2$ or the accuracy score (classification problems) functions can
|
||||
@@ -145,7 +145,7 @@ also be utilized in the present analysis.
|
||||
The data sets that we propose here are (the default sets)
|
||||
|
||||
\begin{itemize}
|
||||
\item Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be
|
||||
\item Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
|
||||
\begin{enumerate}
|
||||
|
||||
\item Either the Franke function or the terrain data from project 1, or data sets your propose.
|
||||
@@ -153,7 +153,7 @@ The data sets that we propose here are (the default sets)
|
||||
\end{enumerate}
|
||||
|
||||
\noindent
|
||||
\item Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine.
|
||||
\item Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called \href{{https://en.wikipedia.org/wiki/MNIST_database}}{MNIST} data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html}}{week 41}
|
||||
\end{itemize}
|
||||
|
||||
\noindent
|
||||
@@ -161,105 +161,124 @@ However, if you would like to study other data sets, feel free to
|
||||
propose other sets. What we listed here are mere suggestions from our
|
||||
side. If you opt for another data set, consider using a set which
|
||||
has been studied in the scientific literature. This makes it easier
|
||||
for you to compare and analyze your results. It is also an essential
|
||||
elements of the scientific discussion.
|
||||
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
|
||||
element of the scientific discussion.
|
||||
|
||||
In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html}}{Scikit-Learn site} or at the \href{{https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)}}{University of California at Irvine}. The \href{{https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html}}{lecture slides on dimensionality reduction have several code examples on this data set}.
|
||||
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html}}{Scikit-Learn site} or at the \href{{https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)}}{University of California at Irvine}.
|
||||
|
||||
\paragraph{Part a): Write your Logistic Regression code, first step.}
|
||||
If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article.
|
||||
We recommend also that you study the code example in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html}}{Logistic Regression}.
|
||||
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
|
||||
|
||||
Write the part of the code which reads in the data and sets up the relevant data sets.
|
||||
\paragraph{Part a): Write your own Stochastic Gradient Descent code, first step.}
|
||||
In order to get started, we will now replace in our standard ordinary
|
||||
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
|
||||
algorithm with our own SGD code. You can choose whether you want to
|
||||
add the momentum SGD optionality or other SGD variants such as RMSprop
|
||||
or ADAgrad. The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html}}{week 40 contain more
|
||||
details}
|
||||
|
||||
\paragraph{Part b): Write your Logistic Regression code, second step.}
|
||||
Now you should write your Logistic Regression code with the aim to
|
||||
reproduce the Logistic Regression analysis of the \href{{https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf}}{scientific
|
||||
article}.
|
||||
Perform an analysis of the results for OLS and Ridge regression as
|
||||
function of the chosen learning rates, the number of mini-batches and
|
||||
epochs as well as algorithm for scaling the learning rate. You can
|
||||
also compare your own results with those that can be obtained using
|
||||
for example \textbf{Scikit-Learn}'s various SGD options. Discuss your
|
||||
results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and
|
||||
the learning rate $\gamma$. Discuss your results.
|
||||
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
|
||||
|
||||
In order to find the optimal parameters of your logistic regressor you
|
||||
should include a gradient descent solver, as discussed in the
|
||||
\href{{https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html}}{gradient descent
|
||||
lectures}.
|
||||
Since we don't have so many data points, you may just code the
|
||||
standard gradient descent with a given learning rate, or even attempt
|
||||
to use the Newton-Raphson method. Alternatively, it may be useful for
|
||||
the next part on neural networks to implement a stochastic gradient
|
||||
descent with and without mini-batches. Stochastic gradient with
|
||||
mini-batches may give the best results. You could finally compare your
|
||||
code with the output from \textbf{scikit-learn}'s toolbox for optimization
|
||||
methods applied to logistic regression.
|
||||
|
||||
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called \emph{accuracy} score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets $t_i$ divided by the total
|
||||
number of targets. A perfect classifier will have an accuracy score of
|
||||
$1$.
|
||||
|
||||
\[
|
||||
\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} ,
|
||||
\]
|
||||
|
||||
where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$
|
||||
otherwise if we have a binary classifcation problem. Here $t_i$
|
||||
represents the target and $y_i$ the outputs of your Logistic
|
||||
Regression code.
|
||||
|
||||
|
||||
You can compare your own results with those obtained using
|
||||
\textbf{scikit-learn}.
|
||||
|
||||
|
||||
As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \
|
||||
Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \
|
||||
information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn\
|
||||
.datasets.load_breast_cancer.html}}{Scikit-Learn site} or at the "University of California at Irvine":"https://archive.ics.uci.e\
|
||||
du/ml/datasets/breast+cancer+wisconsin+(original)". The \href{{https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\
|
||||
ed.html}}{lecture slides on dimensionality reduction have sev\
|
||||
eral code examples on this data set}.
|
||||
|
||||
|
||||
\paragraph{Part c): Writing your own Neural Network code.}
|
||||
\paragraph{Part b): Writing your own Neural Network code.}
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write to your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html}}{lecture
|
||||
slides}. We
|
||||
start with the Logistic Regression case and the data set discussed in parts a) and b) but train
|
||||
now the network to find the optimal weights and biases. You are free
|
||||
to use the codes in the above lecture slides as starting points.
|
||||
write your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the lecture slides from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html}}{week 41}.
|
||||
|
||||
Discuss again your choice of cost function.
|
||||
We will focus on a regression problem first and study either the
|
||||
Franke function or terrain data (or both or other data sets) from
|
||||
project 1. Discuss again your choice of cost function.
|
||||
|
||||
Train your network and compare the results with those from your Logistic Regression code.
|
||||
You should test your results against a similar code using \textbf{Scikit-Learn} (see the examples in the above lecture notes) or \textbf{tensorflow/keras}.
|
||||
Write an FFNN code for regression with a flexible number of hidden
|
||||
layers and nodes using the Sigmoid function as activation function for
|
||||
the hidden layers. Initialize the weights using a normal
|
||||
distribution. How would you initialize the biases? And which
|
||||
activation function would you select for the final output layer?
|
||||
|
||||
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
|
||||
You should test your results against a similar code using \textbf{Scikit-Learn} (see the examples in the above lecture notes from week 41) or \textbf{tensorflow/keras}.
|
||||
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Logistic Regression code and your own Neural Network
|
||||
code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.
|
||||
obtained with the Linear Regression code and your own Neural Network
|
||||
code. Compare the results with those from project 1.
|
||||
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores.
|
||||
|
||||
A useful reference on the back progagation algorithm is \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's
|
||||
book}. It is an excellent
|
||||
read.
|
||||
|
||||
|
||||
\paragraph{Part d): Regression analysis using neural networks.}
|
||||
|
||||
\paragraph{Part c): Testing different activation functions.}
|
||||
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
|
||||
|
||||
\paragraph{Part d): Classification analysis using neural networks.}
|
||||
With a well-written code it should now be easy to change the
|
||||
activation function for the output layer.
|
||||
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in part c) in order to perform a regression (fitting a
|
||||
function or some data set) analysis. As stated above, our default data
|
||||
sets could be either the Franke function or the terrain data from
|
||||
project 1.
|
||||
developed in parts b) and c) in order to perform a classification analysis.
|
||||
|
||||
Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)
|
||||
with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.
|
||||
We will here study the MNIST data set of hand-written numbers as
|
||||
discussed in the lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html}}{week
|
||||
41}. Use
|
||||
the \textbf{Softmax} function as activation function. Your code should
|
||||
however also be able to use a binary activation function as well.
|
||||
|
||||
Again, we strongly recommend that you compare your own neural Network code and results against a similar code using \textbf{Scikit-Learn} (see the examples in the above lecture notes) or \textbf{tensorflow/keras}.
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called \emph{accuracy} score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets $t_i$ divided by the total
|
||||
number of targets, that is
|
||||
|
||||
|
||||
\[
|
||||
\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} ,
|
||||
\]
|
||||
|
||||
where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$
|
||||
otherwise if we have a binary classification problem. Here $t_i$
|
||||
represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.
|
||||
|
||||
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\lambda$ (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
|
||||
|
||||
|
||||
As stated in the introduction, it can also be useful to study other
|
||||
datasets. In particular, the so-called Wisconsin Cancer
|
||||
data (which is a binary problem, benign or malignant tumors) may be
|
||||
studied. You find more information about this at the \href{{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html}}{Scikit-Learn
|
||||
site} or at the \href{{https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)}}{University of California
|
||||
at Irvine}.
|
||||
|
||||
|
||||
|
||||
\paragraph{Part e) Critical evaluation of the various algorithms.}
|
||||
|
||||
Again, we strongly recommend that you compare your own neural Network
|
||||
code for classification and pertinent results against a similar code using \textbf{Scikit-Learn} or \textbf{tensorflow/keras} or \textbf{pytorch}.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
\paragraph{Part e): Write your Logistic Regression code, final step.}
|
||||
Finally, we want to compare the FFNN code we have developed with
|
||||
Logistic regression, that is we wish to compare our neural network
|
||||
classification results with the results we can obtain with another
|
||||
method.
|
||||
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.
|
||||
Add also an $l_2$ regularization parameter $\lambda$. Compare your results with those from your FFNN code as well as those obtained using \textbf{Scikit-Learn}'s logistic regression functionality.
|
||||
|
||||
The weblink here \href{{https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3}}{\nolinkurl{https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3}}compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
|
||||
|
||||
|
||||
\paragraph{Part f) Critical evaluation of the various algorithms.}
|
||||
After all these glorious calculations, you should now summarize the
|
||||
various algorithms and come with a critical evaluation of their pros
|
||||
and cons. Which algorithm works best for the regression case and which
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
TITLE: Project 2 on Machine Learning, deadline November 7
|
||||
TITLE: Project 2 on Machine Learning, deadline November 9
|
||||
AUTHOR: "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo, Norway
|
||||
DATE: today
|
||||
|
||||
@@ -7,8 +7,8 @@ DATE: today
|
||||
|
||||
The main aim of this project is to study both classification and
|
||||
regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied
|
||||
in project 1. We will include logistic regresion for classification
|
||||
problems and write our own FFNNcode for studying
|
||||
in project 1. We will also include logistic regression for classification
|
||||
problems and write our own FFNN code for studying
|
||||
both regression and classification problems. The codes developed in
|
||||
project 1, including bootstrap and/or cross-validation as well as the
|
||||
computation of the mean-squared error and/or the $R2$ or the accuracy score (classification problems) functions can
|
||||
@@ -17,53 +17,95 @@ also be utilized in the present analysis.
|
||||
|
||||
The data sets that we propose here are (the default sets)
|
||||
|
||||
* Regression (fitting a continuous function). In this part you will need to bring up your results from project 1 and compare these with what you get from you Neural Network code to be developed here. The data sets could be
|
||||
* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be
|
||||
o Either the Franke function or the terrain data from project 1, or data sets your propose.
|
||||
* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called MNIST data set of images representing hand-written numbers from zero to nine.
|
||||
* Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called "MNIST":"https://en.wikipedia.org/wiki/MNIST_database" data set of images representing hand-written numbers from zero to nine. These are discussed intensively in the lecture notes on neural networks, see for example the slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html"
|
||||
|
||||
However, if you would like to study other data sets, feel free to
|
||||
propose other sets. What we listed here are mere suggestions from our
|
||||
side. If you opt for another data set, consider using a set which
|
||||
has been studied in the scientific literature. This makes it easier
|
||||
for you to compare and analyze your results. It is also an essential
|
||||
elements of the scientific discussion.
|
||||
for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential
|
||||
element of the scientific discussion.
|
||||
|
||||
In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)". The "lecture slides on dimensionality reduction have several code examples on this data set":"https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimRed.html".
|
||||
In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)".
|
||||
|
||||
=== Part a): Write your Logistic Regression code, first step ===
|
||||
We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code.
|
||||
|
||||
If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article.
|
||||
We recommend also that you study the code example in the "Logistic Regression":"https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html".
|
||||
=== Part a): Write your own Stochastic Gradient Descent code, first step ===
|
||||
|
||||
Write the part of the code which reads in the data and sets up the relevant data sets.
|
||||
In order to get started, we will now replace in our standard ordinary
|
||||
least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion
|
||||
algorithm with our own SGD code. You can choose whether you want to
|
||||
add the momentum SGD optionality or other SGD variants such as RMSprop
|
||||
or ADAgrad. The lecture notes from "week 40 contain more
|
||||
details":"https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html"
|
||||
|
||||
=== Part b): Write your Logistic Regression code, second step ===
|
||||
Perform an analysis of the results for OLS and Ridge regression as
|
||||
function of the chosen learning rates, the number of mini-batches and
|
||||
epochs as well as algorithm for scaling the learning rate. You can
|
||||
also compare your own results with those that can be obtained using
|
||||
for example _Scikit-Learn_'s various SGD options. Discuss your
|
||||
results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and
|
||||
the learning rate $\gamma$. Discuss your results.
|
||||
|
||||
Now you should write your Logistic Regression code with the aim to
|
||||
reproduce the Logistic Regression analysis of the "scientific
|
||||
article":"https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf".
|
||||
You will need your SGD code for the setup of the Neural Network and Logistic Regression codes.
|
||||
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
=== Part b): Writing your own Neural Network code ===
|
||||
|
||||
In order to find the optimal parameters of your logistic regressor you
|
||||
should include a gradient descent solver, as discussed in the
|
||||
"gradient descent
|
||||
lectures":"https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html".
|
||||
Since we don't have so many data points, you may just code the
|
||||
standard gradient descent with a given learning rate, or even attempt
|
||||
to use the Newton-Raphson method. Alternatively, it may be useful for
|
||||
the next part on neural networks to implement a stochastic gradient
|
||||
descent with and without mini-batches. Stochastic gradient with
|
||||
mini-batches may give the best results. You could finally compare your
|
||||
code with the output from _scikit-learn_'s toolbox for optimization
|
||||
methods applied to logistic regression.
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the lecture slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html".
|
||||
|
||||
We will focus on a regression problem first and study either the
|
||||
Franke function or terrain data (or both or other data sets) from
|
||||
project 1. Discuss again your choice of cost function.
|
||||
|
||||
Write an FFNN code for regression with a flexible number of hidden
|
||||
layers and nodes using the Sigmoid function as activation function for
|
||||
the hidden layers. Initialize the weights using a normal
|
||||
distribution. How would you initialize the biases? And which
|
||||
activation function would you select for the final output layer?
|
||||
|
||||
Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1.
|
||||
You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes from week 41) or _tensorflow/keras_.
|
||||
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Linear Regression code and your own Neural Network
|
||||
code. Compare the results with those from project 1.
|
||||
Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores.
|
||||
|
||||
A useful reference on the back progagation algorithm is "Nielsen's
|
||||
book":"http://neuralnetworksanddeeplearning.com/". It is an excellent
|
||||
read.
|
||||
|
||||
|
||||
|
||||
=== Part c): Testing different activation functions ===
|
||||
|
||||
You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases.
|
||||
|
||||
=== Part d): Classification analysis using neural networks ===
|
||||
|
||||
|
||||
|
||||
With a well-written code it should now be easy to change the
|
||||
activation function for the output layer.
|
||||
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in parts b) and c) in order to perform a classification analysis.
|
||||
|
||||
We will here study the MNIST data set of hand-written numbers as
|
||||
discussed in the lecture notes from "week
|
||||
41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html". Use
|
||||
the _Softmax_ function as activation function. Your code should
|
||||
however also be able to use a binary activation function as well.
|
||||
|
||||
To measure the performance of our classification problem we use the
|
||||
so-called *accuracy* score. The accuracy is as you would expect just
|
||||
the number of correctly guessed targets $t_i$ divided by the total
|
||||
number of targets. A perfect classifier will have an accuracy score of
|
||||
$1$.
|
||||
number of targets, that is
|
||||
|
||||
|
||||
!bt
|
||||
\[
|
||||
@@ -72,64 +114,44 @@ $1$.
|
||||
!et
|
||||
|
||||
where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$
|
||||
otherwise if we have a binary classifcation problem. Here $t_i$
|
||||
represents the target and $y_i$ the outputs of your Logistic
|
||||
Regression code.
|
||||
otherwise if we have a binary classification problem. Here $t_i$
|
||||
represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.
|
||||
|
||||
Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\lambda$ (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
|
||||
|
||||
|
||||
You can compare your own results with those obtained using
|
||||
_scikit-learn_.
|
||||
|
||||
|
||||
As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \
|
||||
Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \
|
||||
information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn\
|
||||
.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.e\
|
||||
du/ml/datasets/breast+cancer+wisconsin+(original)". The "lecture slides on dimensionality reduction have sev\
|
||||
eral code examples on this data set":"https://compphysics.github.io/MachineLearning/doc/pub/DimRed/html/DimR\
|
||||
ed.html".
|
||||
|
||||
|
||||
=== Part c): Writing your own Neural Network code ===
|
||||
|
||||
Your aim now, and this is the central part of this project, is to
|
||||
write to your own Feed Forward Neural Network code implementing the back
|
||||
propagation algorithm discussed in the "lecture
|
||||
slides":"https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html". We
|
||||
start with the Logistic Regression case and the data set discussed in parts a) and b) but train
|
||||
now the network to find the optimal weights and biases. You are free
|
||||
to use the codes in the above lecture slides as starting points.
|
||||
|
||||
Discuss again your choice of cost function.
|
||||
|
||||
Train your network and compare the results with those from your Logistic Regression code.
|
||||
You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes) or _tensorflow/keras_.
|
||||
|
||||
Comment your results and give a critical discussion of the results
|
||||
obtained with the Logistic Regression code and your own Neural Network
|
||||
code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score.
|
||||
|
||||
A useful reference on the back progagation algorithm is "Nielsen's
|
||||
book":"http://neuralnetworksanddeeplearning.com/". It is an excellent
|
||||
read.
|
||||
|
||||
|
||||
=== Part d): Regression analysis using neural networks ===
|
||||
|
||||
Here we will change the cost function for our neural network code
|
||||
developed in part c) in order to perform a regression (fitting a
|
||||
function or some data set) analysis. As stated above, our default data
|
||||
sets could be either the Franke function or the terrain data from
|
||||
project 1.
|
||||
|
||||
Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters)
|
||||
with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1.
|
||||
|
||||
Again, we strongly recommend that you compare your own neural Network code and results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes) or _tensorflow/keras_.
|
||||
As stated in the introduction, it can also be useful to study other
|
||||
datasets. In particular, the so-called Wisconsin Cancer
|
||||
data (which is a binary problem, benign or malignant tumors) may be
|
||||
studied. You find more information about this at the "Scikit-Learn
|
||||
site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California
|
||||
at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)".
|
||||
|
||||
|
||||
|
||||
=== Part e) Critical evaluation of the various algorithms ===
|
||||
|
||||
Again, we strongly recommend that you compare your own neural Network
|
||||
code for classification and pertinent results against a similar code using _Scikit-Learn_ or _tensorflow/keras_ or _pytorch_.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
=== Part e): Write your Logistic Regression code, final step ===
|
||||
|
||||
Finally, we want to compare the FFNN code we have developed with
|
||||
Logistic regression, that is we wish to compare our neural network
|
||||
classification results with the results we can obtain with another
|
||||
method.
|
||||
|
||||
Define your cost function and the design matrix before you start writing your code.
|
||||
Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates.
|
||||
Add also an $l_2$ regularization parameter $\lambda$. Compare your results with those from your FFNN code as well as those obtained using _Scikit-Learn_'s logistic regression functionality.
|
||||
|
||||
The weblink here URL:"https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3"compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article.
|
||||
|
||||
|
||||
=== Part f) Critical evaluation of the various algorithms ===
|
||||
|
||||
After all these glorious calculations, you should now summarize the
|
||||
various algorithms and come with a critical evaluation of their pros
|
||||
|
||||
Reference in New Issue
Block a user