@@ -153,7 +159,7 @@ MathJax.Hub.Config({
-
The main aim of this project is to study both classification and regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied -in project 1. We will include logistic regresion for classification -problems and write our own FFNNcode for studying +in project 1. We will also include logistic regression for classification +problems and write our own FFNN code for studying both regression and classification problems. The codes developed in project 1, including bootstrap and/or cross-validation as well as the computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can @@ -174,64 +180,111 @@ also be utilized in the present analysis. The data sets that we propose here are (the default sets)
-In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the Scikit-Learn site or at the University of California at Irvine. The lecture slides on dimensionality reduction have several code examples on this data set. - -
-If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article. -We recommend also that you study the code example in the Logistic Regression. +We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code. + +
-Write the part of the code which reads in the data and sets up the relevant data sets. - -
-Now you should write your Logistic Regression code with the aim to -reproduce the Logistic Regression analysis of the scientific -article. +Perform an analysis of the results for OLS and Ridge regression as +function of the chosen learning rates, the number of mini-batches and +epochs as well as algorithm for scaling the learning rate. You can +also compare your own results with those that can be obtained using +for example Scikit-Learn's various SGD options. Discuss your +results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and +the learning rate \( \gamma \). Discuss your results.
-Define your cost function and the design matrix before you start writing your code. +You will need your SGD code for the setup of the Neural Network and Logistic Regression codes. + +
-In order to find the optimal parameters of your logistic regressor you -should include a gradient descent solver, as discussed in the -gradient descent -lectures. -Since we don't have so many data points, you may just code the -standard gradient descent with a given learning rate, or even attempt -to use the Newton-Raphson method. Alternatively, it may be useful for -the next part on neural networks to implement a stochastic gradient -descent with and without mini-batches. Stochastic gradient with -mini-batches may give the best results. You could finally compare your -code with the output from scikit-learn's toolbox for optimization -methods applied to logistic regression. +Your aim now, and this is the central part of this project, is to +write your own Feed Forward Neural Network code implementing the back +propagation algorithm discussed in the lecture slides from week 41. + +
+We will focus on a regression problem first and study either the +Franke function or terrain data (or both or other data sets) from +project 1. Discuss again your choice of cost function. + +
+Write an FFNN code for regression with a flexible number of hidden +layers and nodes using the Sigmoid function as activation function for +the hidden layers. Initialize the weights using a normal +distribution. How would you initialize the biases? And which +activation function would you select for the final output layer? + +
+Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1. +You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes from week 41) or tensorflow/keras. + +
+Comment your results and give a critical discussion of the results +obtained with the Linear Regression code and your own Neural Network +code. Compare the results with those from project 1. +Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores. + +
+A useful reference on the back progagation algorithm is Nielsen's +book. It is an excellent +read. + +
+You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases. + +
+With a well-written code it should now be easy to change the +activation function for the output layer. + +
+Here we will change the cost function for our neural network code +developed in parts b) and c) in order to perform a classification analysis. + +
+We will here study the MNIST data set of hand-written numbers as +discussed in the lecture notes from week +41. Use +the Softmax function as activation function. Your code should +however also be able to use a binary activation function as well.
To measure the performance of our classification problem we use the so-called accuracy score. The accuracy is as you would expect just the number of correctly guessed targets \( t_i \) divided by the total -number of targets. A perfect classifier will have an accuracy score of -\( 1 \). +number of targets, that is $$ \text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , @@ -239,68 +292,41 @@ $$
where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \) -otherwise if we have a binary classifcation problem. Here \( t_i \) -represents the target and \( y_i \) the outputs of your Logistic -Regression code. +otherwise if we have a binary classification problem. Here \( t_i \) +represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).
-You can compare your own results with those obtained using -scikit-learn. +Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
-As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \ -Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \ -information about this at the Scikit-Learn site or at the "University of California at Irvine":"https://archive.ics.uci.e\ -du/ml/datasets/breast+cancer+wisconsin+(original)". The lecture slides on dimensionality reduction have sev\ -eral code examples on this data set. - -
-Your aim now, and this is the central part of this project, is to -write to your own Feed Forward Neural Network code implementing the back -propagation algorithm discussed in the lecture -slides. We -start with the Logistic Regression case and the data set discussed in parts a) and b) but train -now the network to find the optimal weights and biases. You are free -to use the codes in the above lecture slides as starting points. +Again, we strongly recommend that you compare your own neural Network +code for classification and pertinent results against a similar code using Scikit-Learn or tensorflow/keras or pytorch. + +
-Discuss again your choice of cost function. +Finally, we want to compare the FFNN code we have developed with +Logistic regression, that is we wish to compare our neural network +classification results with the results we can obtain with another +method.
-Train your network and compare the results with those from your Logistic Regression code. -You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. +Define your cost function and the design matrix before you start writing your code. +Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates. +Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using Scikit-Learn's logistic regression functionality.
-Comment your results and give a critical discussion of the results -obtained with the Logistic Regression code and your own Neural Network -code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score. +The weblink here https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article. -
-A useful reference on the back progagation algorithm is Nielsen's -book. It is an excellent -read. - -
-Here we will change the cost function for our neural network code -developed in part c) in order to perform a regression (fitting a -function or some data set) analysis. As stated above, our default data -sets could be either the Franke function or the terrain data from -project 1. - -
-Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters) -with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1. - -
-Again, we strongly recommend that you compare your own neural Network code and results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. - -
After all these glorious calculations, you should now summarize the @@ -309,7 +335,7 @@ and cons. Which algorithm works best for the regression case and which is best for the classification case. These codes can also be part of your final project 3, but now applied to other data sets. -
Here follows a brief recipe and recommendation on how to write a report for each @@ -335,7 +361,7 @@ project.
The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report: @@ -350,7 +376,7 @@ Finally, we encourage you to collaborate. Optimal working groups consist of 2-3 students. You can then hand in a common report. -
If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, diff --git a/doc/Projects/2020/Project2/html/Project2-bs.html b/doc/Projects/2020/Project2/html/Project2-bs.html index 4ccada940..0b41c7ed5 100644 --- a/doc/Projects/2020/Project2/html/Project2-bs.html +++ b/doc/Projects/2020/Project2/html/Project2-bs.html @@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source - + -
@@ -153,7 +159,7 @@ MathJax.Hub.Config({
-
The main aim of this project is to study both classification and regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied -in project 1. We will include logistic regresion for classification -problems and write our own FFNNcode for studying +in project 1. We will also include logistic regression for classification +problems and write our own FFNN code for studying both regression and classification problems. The codes developed in project 1, including bootstrap and/or cross-validation as well as the computation of the mean-squared error and/or the \( R2 \) or the accuracy score (classification problems) functions can @@ -174,64 +180,111 @@ also be utilized in the present analysis. The data sets that we propose here are (the default sets)
-In particular, when developing your own Logistic Regression code for classification problems, the so-called Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more information about this at the Scikit-Learn site or at the University of California at Irvine. The lecture slides on dimensionality reduction have several code examples on this data set. - -
-If you opt for the credit card data, your first task is to familiarize yourself with the data set and the scientific article. -We recommend also that you study the code example in the Logistic Regression. +We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code. + +
-Write the part of the code which reads in the data and sets up the relevant data sets. - -
-Now you should write your Logistic Regression code with the aim to -reproduce the Logistic Regression analysis of the scientific -article. +Perform an analysis of the results for OLS and Ridge regression as +function of the chosen learning rates, the number of mini-batches and +epochs as well as algorithm for scaling the learning rate. You can +also compare your own results with those that can be obtained using +for example Scikit-Learn's various SGD options. Discuss your +results. For Ridge regression you need now to study the results as functions of the hyper-parameter \( \lambda \) and +the learning rate \( \gamma \). Discuss your results.
-Define your cost function and the design matrix before you start writing your code. +You will need your SGD code for the setup of the Neural Network and Logistic Regression codes. + +
-In order to find the optimal parameters of your logistic regressor you -should include a gradient descent solver, as discussed in the -gradient descent -lectures. -Since we don't have so many data points, you may just code the -standard gradient descent with a given learning rate, or even attempt -to use the Newton-Raphson method. Alternatively, it may be useful for -the next part on neural networks to implement a stochastic gradient -descent with and without mini-batches. Stochastic gradient with -mini-batches may give the best results. You could finally compare your -code with the output from scikit-learn's toolbox for optimization -methods applied to logistic regression. +Your aim now, and this is the central part of this project, is to +write your own Feed Forward Neural Network code implementing the back +propagation algorithm discussed in the lecture slides from week 41. + +
+We will focus on a regression problem first and study either the +Franke function or terrain data (or both or other data sets) from +project 1. Discuss again your choice of cost function. + +
+Write an FFNN code for regression with a flexible number of hidden +layers and nodes using the Sigmoid function as activation function for +the hidden layers. Initialize the weights using a normal +distribution. How would you initialize the biases? And which +activation function would you select for the final output layer? + +
+Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1. +You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes from week 41) or tensorflow/keras. + +
+Comment your results and give a critical discussion of the results +obtained with the Linear Regression code and your own Neural Network +code. Compare the results with those from project 1. +Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and \( R2 \) scores. + +
+A useful reference on the back progagation algorithm is Nielsen's +book. It is an excellent +read. + +
+You should now also test different activation functions for the hidden layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and discuss your results. You may also study the way you initialize your weights and biases. + +
+With a well-written code it should now be easy to change the +activation function for the output layer. + +
+Here we will change the cost function for our neural network code +developed in parts b) and c) in order to perform a classification analysis. + +
+We will here study the MNIST data set of hand-written numbers as +discussed in the lecture notes from week +41. Use +the Softmax function as activation function. Your code should +however also be able to use a binary activation function as well.
To measure the performance of our classification problem we use the so-called accuracy score. The accuracy is as you would expect just the number of correctly guessed targets \( t_i \) divided by the total -number of targets. A perfect classifier will have an accuracy score of -\( 1 \). +number of targets, that is $$ \text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , @@ -239,68 +292,41 @@ $$
where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \) -otherwise if we have a binary classifcation problem. Here \( t_i \) -represents the target and \( y_i \) the outputs of your Logistic -Regression code. +otherwise if we have a binary classification problem. Here \( t_i \) +represents the target and \( y_i \) the outputs of your FFNN code and \( n \) is simply the number of targets \( t_i \).
-You can compare your own results with those obtained using -scikit-learn. +Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \( \lambda \) (as you did in Ridge Regression), various activation functions, number of hidden layers and nodes and activation functions.
-As stated in the introduction, it can also be useful to study other datasets. In particular, when developing your own Logistic Regression code for classification problems, the so-called \ -Wisconsin Cancer data (which is a binary problem, benign or malignant tumors) may be studied. You find more \ -information about this at the Scikit-Learn site or at the "University of California at Irvine":"https://archive.ics.uci.e\ -du/ml/datasets/breast+cancer+wisconsin+(original)". The lecture slides on dimensionality reduction have sev\ -eral code examples on this data set. - -
-Your aim now, and this is the central part of this project, is to -write to your own Feed Forward Neural Network code implementing the back -propagation algorithm discussed in the lecture -slides. We -start with the Logistic Regression case and the data set discussed in parts a) and b) but train -now the network to find the optimal weights and biases. You are free -to use the codes in the above lecture slides as starting points. +Again, we strongly recommend that you compare your own neural Network +code for classification and pertinent results against a similar code using Scikit-Learn or tensorflow/keras or pytorch. + +
-Discuss again your choice of cost function. +Finally, we want to compare the FFNN code we have developed with +Logistic regression, that is we wish to compare our neural network +classification results with the results we can obtain with another +method.
-Train your network and compare the results with those from your Logistic Regression code. -You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. +Define your cost function and the design matrix before you start writing your code. +Write thereafter a Logistic regression code using your SGD algorithm. Study the results as functions of the chosen learning rates. +Add also an \( l_2 \) regularization parameter \( \lambda \). Compare your results with those from your FFNN code as well as those obtained using Scikit-Learn's logistic regression functionality.
-Comment your results and give a critical discussion of the results -obtained with the Logistic Regression code and your own Neural Network -code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal accurary score. +The weblink here https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3compares logistic regression and FFNN using the MNIST data set. You may find several useful hints and ideas from this article. -
-A useful reference on the back progagation algorithm is Nielsen's -book. It is an excellent -read. - -
-Here we will change the cost function for our neural network code -developed in part c) in order to perform a regression (fitting a -function or some data set) analysis. As stated above, our default data -sets could be either the Franke function or the terrain data from -project 1. - -
-Compare you results from the neural network regression analysis (with a discussion of learning rates and regularization parameters) -with those you obtained in project 1. Alternatively, if you opt for other data sets, you would need to run your standard ordinary least squares, Ridge and Lasso calculations using your codes from project 1. - -
-Again, we strongly recommend that you compare your own neural Network code and results against a similar code using Scikit-Learn (see the examples in the above lecture notes) or tensorflow/keras. - -
After all these glorious calculations, you should now summarize the @@ -309,7 +335,7 @@ and cons. Which algorithm works best for the regression case and which is best for the classification case. These codes can also be part of your final project 3, but now applied to other data sets. -
Here follows a brief recipe and recommendation on how to write a report for each @@ -335,7 +361,7 @@ project.
The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report: @@ -350,7 +376,7 @@ Finally, we encourage you to collaborate. Optimal working groups consist of 2-3 students. You can then hand in a common report. -
If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, diff --git a/doc/Projects/2020/Project2/html/Project2.html b/doc/Projects/2020/Project2/html/Project2.html index ac2cc35e3..474c8a8ee 100644 --- a/doc/Projects/2020/Project2/html/Project2.html +++ b/doc/Projects/2020/Project2/html/Project2.html @@ -7,9 +7,9 @@ Automatically generated HTML file from DocOnce source - + -