From b2df29a7951019b696e0ac84fd28a8f09b92efac Mon Sep 17 00:00:00 2001 From: mhjensen Date: Wed, 7 Oct 2020 07:08:08 +0200 Subject: [PATCH] update p2 --- .../2020/Project2/html/._Project2-bs000.html | 242 ++++++++++-------- .../2020/Project2/html/Project2-bs.html | 242 ++++++++++-------- doc/Projects/2020/Project2/html/Project2.html | 221 +++++++++------- .../2020/Project2/ipynb/Project2.ipynb | 191 +++++++------- .../Project2/ipynb/ipynb-Project2-src.tar.gz | Bin 192 -> 193 bytes doc/Projects/2020/Project2/pdf/Project2.p.tex | 183 +++++++------ doc/Projects/2020/Project2/pdf/Project2.pdf | Bin 236422 -> 241271 bytes doc/Projects/2020/Project2/pdf/Project2.tex | 183 +++++++------ .../Projects/2020/Project2/Project2.do.txt | 190 ++++++++------ 9 files changed, 804 insertions(+), 648 deletions(-) diff --git a/doc/Projects/2020/Project2/html/._Project2-bs000.html b/doc/Projects/2020/Project2/html/._Project2-bs000.html index 4ccada940..0b41c7ed5 100644 --- a/doc/Projects/2020/Project2/html/._Project2-bs000.html +++ b/doc/Projects/2020/Project2/html/._Project2-bs000.html @@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source - + -Project 2 on Machine Learning, deadline November 7 +Project 2 on Machine Learning, deadline November 9 @@ -46,33 +46,38 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec0'), - ('Part a): Write your Logistic Regression code, first step', + ('Part a): Write your own Stochastic Gradient Descent code, ' + 'first step', 3, None, '___sec1'), - ('Part b): Write your Logistic Regression code, second step', + ('Part b): Writing your own Neural Network code', 3, None, '___sec2'), - ('Part c): Writing your own Neural Network code', + ('Part c): Testing different activation functions', 3, None, '___sec3'), - ('Part d): Regression analysis using neural networks', + ('Part d): Classification analysis using neural networks', 3, None, '___sec4'), - ('Part e) Critical evaluation of the various algorithms', + ('Part e): Write your Logistic Regression code, final step', 3, None, '___sec5'), - ('Background literature', 2, None, '___sec6'), - ('Introduction to numerical projects', 2, None, '___sec7'), + ('Part f) Critical evaluation of the various algorithms', + 3, + None, + '___sec6'), + ('Background literature', 2, None, '___sec7'), + ('Introduction to numerical projects', 2, None, '___sec8'), ('Format for electronic delivery of report and programs', 2, None, - '___sec8'), - ('Software and needed installations', 2, None, '___sec9')]} + '___sec9'), + ('Software and needed installations', 2, None, '___sec10')]} end of tocinfo --> @@ -102,7 +107,7 @@ MathJax.Hub.Config({ - Project 2 on Machine Learning, deadline November 7 + Project 2 on Machine Learning, deadline November 9
@@ -111,15 +116,16 @@ MathJax.Hub.Config({ Contents @@ -138,7 +144,7 @@ MathJax.Hub.Config({
-

Project 2 on Machine Learning, deadline November 7

+

Project 2 on Machine Learning, deadline November 9

@@ -153,7 +159,7 @@ MathJax.Hub.Config({

Department of Physics, University of Oslo, Norway

-

Oct 6, 2020

+

Oct 7, 2020


@@ -163,8 +169,8 @@ MathJax.Hub.Config({

The main aim of this project is to study both classification and regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied -in project 1. We will include logistic regresion for classification -problems and write our own FFNNcode for studying +in project 1. We will also include logistic regression for classification +problems and write our own FFNN code for studying both regression and classification problems. The codes developed in project 1, including bootstrap and/or cross-validation as well as the computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can @@ -174,64 +180,111 @@ also be utilized in the present analysis. The data sets that we propose here are (the default sets)

However, if you would like to study other data sets, feel free to propose other sets. What we listed here are mere suggestions from our side. If you opt for another data set, consider using a set which has been studied in the scientific literature. This makes it easier -for you to compare and analyze your results. It is also an essential -elements of the scientific discussion. +for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential +element of the scientific discussion.

-In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the Scikit-Learn site or at the University of California at Irvine. The lecture slides on dimensionality reduction have several code examples on this data set. - -

Part a): Write your Logistic Regression code, first step

+In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the Scikit-Learn site or at the University of California at Irvine.

-If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article. -We recommend also that you study the code example in the Logistic Regression. +We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code. + +

Part a): Write your own Stochastic Gradient Descent code, first step

-Write the part of the code which reads in the data and sets up the relevant data sets. - -

Part b): Write your Logistic Regression code, second step

+In order to get started, we will now replace in our standard ordinary +least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion +algorithm with our own SGD code. You can choose whether you want to +add the momentum SGD optionality or other SGD variants such as RMSprop +or ADAgrad. The lecture notes from week 40 contain more +details

-Now you should write your Logistic Regression code with the aim to -reproduce the Logistic Regression analysis of the scientific -article. +Perform an analysis of the results for OLS and Ridge regression as +function of the chosen learning rates, the number of mini-batches and +epochs as well as algorithm for scaling the learning rate. You can +also compare your own results with those that can be obtained using +for example Scikit-Learn's various SGD options. Discuss your +results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and +the learning rate \( \gamma \). Discuss your results.

-Define your cost function and the design matrix before you start writing your code. +You will need your SGD code for the setup of the Neural Network and Logistic Regression codes. + +

Part b): Writing your own Neural Network code

-In order to find the optimal parameters of your logistic regressor you -should include a gradient descent solver, as discussed in the -gradient descent -lectures. -Since we don't have so many data points, you may just code the -standard gradient descent with a given learning rate, or even attempt -to use the Newton-Raphson method. Alternatively, it may be useful for -the next part on neural networks to implement a stochastic gradient -descent with and without mini-batches. Stochastic gradient with -mini-batches may give the best results. You could finally compare your -code with the output from scikit-learn's toolbox for optimization -methods applied to logistic regression. +Your aim now, and this is the central part of this project, is to +write your own Feed Forward Neural Network code implementing the back +propagation algorithm discussed in the lecture slides from week 41. + +

+We will focus on a regression problem first and study either the +Franke function or terrain data (or both or other data sets) from +project 1. Discuss again your choice of cost function. + +

+Write an FFNN code for regression with a flexible number of hidden +layers and nodes using the Sigmoid function as activation function for +the hidden layers. Initialize the weights using a normal +distribution. How would you initialize the biases? And which +activation function would you select for the final output layer? + +

+Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1. +You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes from week 41) or tensorflow/keras. + +

+Comment your results and give a critical discussion of the results +obtained with the Linear Regression code and your own Neural Network +code. Compare the results with those from project 1. +Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores. + +

+A useful reference on the back progagation algorithm is Nielsen's +book. It is an excellent +read. + +

Part c): Testing different activation functions

+ +

+You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases. + +

Part d): Classification analysis using neural networks

+ +

+With a well-written code it should now be easy to change the +activation function for the output layer. + +

+Here we will change the cost function for our neural network code +developed in parts b) and c) in order to perform a classification analysis. + +

+We will here study the MNIST data set of hand-written numbers as +discussed in the lecture notes from week +41. Use +the Softmax function as activation function. Your code should +however also be able to use a binary activation function as well.

To measure the performance of our classification problem we use the so-called accuracy score. The accuracy is as you would expect just the number of correctly guessed targets \( t_i \) divided by the total -number of targets. A perfect classifier will have an accuracy score of -\( 1 \). +number of targets, that is $$ \text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , @@ -239,68 +292,41 @@ $$

where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \) -otherwise if we have a binary classifcation problem. Here \( t_i \) -represents the target and \( y_i \) the outputs of your Logistic -Regression code. +otherwise if we have a binary classification problem. Here \( t_i \) +represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).

-You can compare your own results with those obtained using -scikit-learn. +Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.

-As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \ -Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \ -information about this at the Scikit-Learn site or at the "University of California at Irvine":"https://archive.ics.uci.e\ -du/ml/datasets/breast+cancer+wisconsin+(original)". The lecture slides on dimensionality reduction have sev\ -eral code examples on this data set. - -

Part c): Writing your own Neural Network code

+As stated in the introduction, it can also be useful to study other +datasets. In particular, the so-called Wisconsin Cancer +data (which is a binary problem, benign or malignant tumors) may be +studied. You find more information about this at the Scikit-Learn +site or at the University of California +at Irvine.

-Your aim now, and this is the central part of this project, is to -write to your own Feed Forward Neural Network code implementing the back -propagation algorithm discussed in the lecture -slides. We -start with the Logistic Regression case and the data set discussed in parts a) and b) but train -now the network to find the optimal weights and biases. You are free -to use the codes in the above lecture slides as starting points. +Again, we strongly recommend that you compare your own neural Network +code for classification and pertinent results against a similar code using Scikit-Learn or tensorflow/keras or pytorch. + +

Part e): Write your Logistic Regression code, final step

-Discuss again your choice of cost function. +Finally, we want to compare the FFNN code we have developed with +Logistic regression, that is we wish to compare our neural network +classification results with the results we can obtain with another +method.

-Train your network and compare the results with those from your Logistic Regression code. -You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. +Define your cost function and the design matrix before you start writing your code. +Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates. +Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using Scikit-Learn's logistic regression functionality.

-Comment your results and give a critical discussion of the results -obtained with the Logistic Regression code and your own Neural Network -code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score. +The weblink here https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article. -

-A useful reference on the back progagation algorithm is Nielsen's -book. It is an excellent -read. - -

Part d): Regression analysis using neural networks

- -

-Here we will change the cost function for our neural network code -developed in part c) in order to perform a regression (fitting a -function or some data set) analysis. As stated above, our default data -sets could be either the Franke function or the terrain data from -project 1. - -

-Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters) -with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1. - -

-Again, we strongly recommend that you compare your own neural Network code and results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. - -

Part e) Critical evaluation of the various algorithms

+

Part f) Critical evaluation of the various algorithms

After all these glorious calculations, you should now summarize the @@ -309,7 +335,7 @@ and cons. Which algorithm works best for the regression case and which is best for the classification case. These codes can also be part of your final project 3, but now applied to other data sets. -

Background literature

+

Background literature

  1. The text of Michael Nielsen is highly recommended, see Nielsen's book. It is an excellent read.
    2. @@ -317,7 +343,7 @@ your final project 3, but now applied to other data sets.
  2. Mehta et al, arXiv 1803.08823, A high-bias, low-variance introduction to Machine Learning for physicists, ArXiv:1803.08823.
-

Introduction to numerical projects

+

Introduction to numerical projects

Here follows a brief recipe and recommendation on how to write a report for each @@ -335,7 +361,7 @@ project.

  • Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.
    • -

    Format for electronic delivery of report and programs

    +

    Format for electronic delivery of report and programs

    The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report: @@ -350,7 +376,7 @@ Finally, we encourage you to collaborate. Optimal working groups consist of 2-3 students. You can then hand in a common report. -

    Software and needed installations

    +

    Software and needed installations

    If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, diff --git a/doc/Projects/2020/Project2/html/Project2-bs.html b/doc/Projects/2020/Project2/html/Project2-bs.html index 4ccada940..0b41c7ed5 100644 --- a/doc/Projects/2020/Project2/html/Project2-bs.html +++ b/doc/Projects/2020/Project2/html/Project2-bs.html @@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source - + -Project 2 on Machine Learning, deadline November 7 +Project 2 on Machine Learning, deadline November 9 @@ -46,33 +46,38 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec0'), - ('Part a): Write your Logistic Regression code, first step', + ('Part a): Write your own Stochastic Gradient Descent code, ' + 'first step', 3, None, '___sec1'), - ('Part b): Write your Logistic Regression code, second step', + ('Part b): Writing your own Neural Network code', 3, None, '___sec2'), - ('Part c): Writing your own Neural Network code', + ('Part c): Testing different activation functions', 3, None, '___sec3'), - ('Part d): Regression analysis using neural networks', + ('Part d): Classification analysis using neural networks', 3, None, '___sec4'), - ('Part e) Critical evaluation of the various algorithms', + ('Part e): Write your Logistic Regression code, final step', 3, None, '___sec5'), - ('Background literature', 2, None, '___sec6'), - ('Introduction to numerical projects', 2, None, '___sec7'), + ('Part f) Critical evaluation of the various algorithms', + 3, + None, + '___sec6'), + ('Background literature', 2, None, '___sec7'), + ('Introduction to numerical projects', 2, None, '___sec8'), ('Format for electronic delivery of report and programs', 2, None, - '___sec8'), - ('Software and needed installations', 2, None, '___sec9')]} + '___sec9'), + ('Software and needed installations', 2, None, '___sec10')]} end of tocinfo --> @@ -102,7 +107,7 @@ MathJax.Hub.Config({ - Project 2 on Machine Learning, deadline November 7 + Project 2 on Machine Learning, deadline November 9

    @@ -111,15 +116,16 @@ MathJax.Hub.Config({ Contents @@ -138,7 +144,7 @@ MathJax.Hub.Config({
    -

    Project 2 on Machine Learning, deadline November 7

    +

    Project 2 on Machine Learning, deadline November 9

    @@ -153,7 +159,7 @@ MathJax.Hub.Config({

    Department of Physics, University of Oslo, Norway

    -

    Oct 6, 2020

    +

    Oct 7, 2020


    @@ -163,8 +169,8 @@ MathJax.Hub.Config({

    The main aim of this project is to study both classification and regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied -in project 1. We will include logistic regresion for classification -problems and write our own FFNNcode for studying +in project 1. We will also include logistic regression for classification +problems and write our own FFNN code for studying both regression and classification problems. The codes developed in project 1, including bootstrap and/or cross-validation as well as the computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can @@ -174,64 +180,111 @@ also be utilized in the present analysis. The data sets that we propose here are (the default sets)

    However, if you would like to study other data sets, feel free to propose other sets. What we listed here are mere suggestions from our side. If you opt for another data set, consider using a set which has been studied in the scientific literature. This makes it easier -for you to compare and analyze your results. It is also an essential -elements of the scientific discussion. +for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential +element of the scientific discussion.

    -In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the Scikit-Learn site or at the University of California at Irvine. The lecture slides on dimensionality reduction have several code examples on this data set. - -

    Part a): Write your Logistic Regression code, first step

    +In particular, when developing your own Neural Network and Logistic Regression codes for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You can find more information about this at the Scikit-Learn site or at the University of California at Irvine.

    -If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article. -We recommend also that you study the code example in the Logistic Regression. +We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code. + +

    Part a): Write your own Stochastic Gradient Descent code, first step

    -Write the part of the code which reads in the data and sets up the relevant data sets. - -

    Part b): Write your Logistic Regression code, second step

    +In order to get started, we will now replace in our standard ordinary +least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion +algorithm with our own SGD code. You can choose whether you want to +add the momentum SGD optionality or other SGD variants such as RMSprop +or ADAgrad. The lecture notes from week 40 contain more +details

    -Now you should write your Logistic Regression code with the aim to -reproduce the Logistic Regression analysis of the scientific -article. +Perform an analysis of the results for OLS and Ridge regression as +function of the chosen learning rates, the number of mini-batches and +epochs as well as algorithm for scaling the learning rate. You can +also compare your own results with those that can be obtained using +for example Scikit-Learn's various SGD options. Discuss your +results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and +the learning rate \( \gamma \). Discuss your results.

    -Define your cost function and the design matrix before you start writing your code. +You will need your SGD code for the setup of the Neural Network and Logistic Regression codes. + +

    Part b): Writing your own Neural Network code

    -In order to find the optimal parameters of your logistic regressor you -should include a gradient descent solver, as discussed in the -gradient descent -lectures. -Since we don't have so many data points, you may just code the -standard gradient descent with a given learning rate, or even attempt -to use the Newton-Raphson method. Alternatively, it may be useful for -the next part on neural networks to implement a stochastic gradient -descent with and without mini-batches. Stochastic gradient with -mini-batches may give the best results. You could finally compare your -code with the output from scikit-learn's toolbox for optimization -methods applied to logistic regression. +Your aim now, and this is the central part of this project, is to +write your own Feed Forward Neural Network code implementing the back +propagation algorithm discussed in the lecture slides from week 41. + +

    +We will focus on a regression problem first and study either the +Franke function or terrain data (or both or other data sets) from +project 1. Discuss again your choice of cost function. + +

    +Write an FFNN code for regression with a flexible number of hidden +layers and nodes using the Sigmoid function as activation function for +the hidden layers. Initialize the weights using a normal +distribution. How would you initialize the biases? And which +activation function would you select for the final output layer? + +

    +Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1. +You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes from week 41) or tensorflow/keras. + +

    +Comment your results and give a critical discussion of the results +obtained with the Linear Regression code and your own Neural Network +code. Compare the results with those from project 1. +Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores. + +

    +A useful reference on the back progagation algorithm is Nielsen's +book. It is an excellent +read. + +

    Part c): Testing different activation functions

    + +

    +You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases. + +

    Part d): Classification analysis using neural networks

    + +

    +With a well-written code it should now be easy to change the +activation function for the output layer. + +

    +Here we will change the cost function for our neural network code +developed in parts b) and c) in order to perform a classification analysis. + +

    +We will here study the MNIST data set of hand-written numbers as +discussed in the lecture notes from week +41. Use +the Softmax function as activation function. Your code should +however also be able to use a binary activation function as well.

    To measure the performance of our classification problem we use the so-called accuracy score. The accuracy is as you would expect just the number of correctly guessed targets \( t_i \) divided by the total -number of targets. A perfect classifier will have an accuracy score of -\( 1 \). +number of targets, that is $$ \text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , @@ -239,68 +292,41 @@ $$

    where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \) -otherwise if we have a binary classifcation problem. Here \( t_i \) -represents the target and \( y_i \) the outputs of your Logistic -Regression code. +otherwise if we have a binary classification problem. Here \( t_i \) +represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).

    -You can compare your own results with those obtained using -scikit-learn. +Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.

    -As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \ -Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \ -information about this at the Scikit-Learn site or at the "University of California at Irvine":"https://archive.ics.uci.e\ -du/ml/datasets/breast+cancer+wisconsin+(original)". The lecture slides on dimensionality reduction have sev\ -eral code examples on this data set. - -

    Part c): Writing your own Neural Network code

    +As stated in the introduction, it can also be useful to study other +datasets. In particular, the so-called Wisconsin Cancer +data (which is a binary problem, benign or malignant tumors) may be +studied. You find more information about this at the Scikit-Learn +site or at the University of California +at Irvine.

    -Your aim now, and this is the central part of this project, is to -write to your own Feed Forward Neural Network code implementing the back -propagation algorithm discussed in the lecture -slides. We -start with the Logistic Regression case and the data set discussed in parts a) and b) but train -now the network to find the optimal weights and biases. You are free -to use the codes in the above lecture slides as starting points. +Again, we strongly recommend that you compare your own neural Network +code for classification and pertinent results against a similar code using Scikit-Learn or tensorflow/keras or pytorch. + +

    Part e): Write your Logistic Regression code, final step

    -Discuss again your choice of cost function. +Finally, we want to compare the FFNN code we have developed with +Logistic regression, that is we wish to compare our neural network +classification results with the results we can obtain with another +method.

    -Train your network and compare the results with those from your Logistic Regression code. -You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. +Define your cost function and the design matrix before you start writing your code. +Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates. +Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using Scikit-Learn's logistic regression functionality.

    -Comment your results and give a critical discussion of the results -obtained with the Logistic Regression code and your own Neural Network -code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score. +The weblink here https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article. -

    -A useful reference on the back progagation algorithm is Nielsen's -book. It is an excellent -read. - -

    Part d): Regression analysis using neural networks

    - -

    -Here we will change the cost function for our neural network code -developed in part c) in order to perform a regression (fitting a -function or some data set) analysis. As stated above, our default data -sets could be either the Franke function or the terrain data from -project 1. - -

    -Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters) -with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1. - -

    -Again, we strongly recommend that you compare your own neural Network code and results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. - -

    Part e) Critical evaluation of the various algorithms

    +

    Part f) Critical evaluation of the various algorithms

    After all these glorious calculations, you should now summarize the @@ -309,7 +335,7 @@ and cons. Which algorithm works best for the regression case and which is best for the classification case. These codes can also be part of your final project 3, but now applied to other data sets. -

    Background literature

    +

    Background literature

    1. The text of Michael Nielsen is highly recommended, see Nielsen's book. It is an excellent read.
      2. @@ -317,7 +343,7 @@ your final project 3, but now applied to other data sets.
    2. Mehta et al, arXiv 1803.08823, A high-bias, low-variance introduction to Machine Learning for physicists, ArXiv:1803.08823.
    -

    Introduction to numerical projects

    +

    Introduction to numerical projects

    Here follows a brief recipe and recommendation on how to write a report for each @@ -335,7 +361,7 @@ project.

  • Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.
    • -

    Format for electronic delivery of report and programs

    +

    Format for electronic delivery of report and programs

    The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report: @@ -350,7 +376,7 @@ Finally, we encourage you to collaborate. Optimal working groups consist of 2-3 students. You can then hand in a common report. -

    Software and needed installations

    +

    Software and needed installations

    If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, diff --git a/doc/Projects/2020/Project2/html/Project2.html b/doc/Projects/2020/Project2/html/Project2.html index ac2cc35e3..474c8a8ee 100644 --- a/doc/Projects/2020/Project2/html/Project2.html +++ b/doc/Projects/2020/Project2/html/Project2.html @@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source - + -Project 2 on Machine Learning, deadline November 7 +Project 2 on Machine Learning, deadline November 9