diff --git a/doc/Projects/2022/Project2/ipynb b/doc/Projects/2022/Project2/ipynb index b59dfb935..42c99dc21 100644 Binary files a/doc/Projects/2022/Project2/ipynb and b/doc/Projects/2022/Project2/ipynb differ diff --git a/doc/Projects/2022/Project2/pdf b/doc/Projects/2022/Project2/pdf index dd29b9c16..ce264c789 100644 Binary files a/doc/Projects/2022/Project2/pdf and b/doc/Projects/2022/Project2/pdf differ diff --git a/doc/src/Projects/2022/Project2/Project2.do.txt b/doc/src/Projects/2022/Project2/Project2.do.txt index 101163b20..217284606 100644 --- a/doc/src/Projects/2022/Project2/Project2.do.txt +++ b/doc/src/Projects/2022/Project2/Project2.do.txt @@ -6,30 +6,37 @@ DATE: today ===== Classification and Regression, from linear and logistic regression to neural networks ===== The main aim of this project is to study both classification and -regression problems by developing our own feed-forward neural network (FFNN) code. We can reuse the regression algorithms studied -in project 1. We will also include logistic regression for classification -problems and write our own FFNN code for studying -both regression and classification problems. The codes developed in -project 1, including bootstrap _and/or_ cross-validation as well as the -computation of the mean-squared error and/or the $R2$ or the accuracy score (classification problems) functions can -also be utilized in the present analysis. +regression problems by developing our own feed-forward neural network +(FFNN) code. We can reuse the regression algorithms studied in project +1. We will also include logistic regression for classification +problems and write our own FFNN code for studying both regression and +classification problems. The codes developed in project 1, including +bootstrap _and/or_ cross-validation as well as the computation of the +mean-squared error and/or the $R2$ or the accuracy score +(classification problems) functions can also be utilized in the +present analysis. -_Important Note_: This project as well as projects 1 and 3 have to be written as a scientific report. The instructions on how to do this and how we grade are available at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md". We will discuss this format during the various lab sessions. Please do spend some time to read our guidelines. The data sets that we propose here are (the default sets) * Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be - o Either the Franke function or the terrain data from project 1, or data sets your propose. It could be a simpler function than the Franke function. + o A simple one-dimensional function or the Franke function or the terrain data from project 1, or data sets your propose. It could be a simpler function than the Franke function. We recommend testing a simpler function (see below). But if you wish to try more complex function, feel free to do so. * Classification. Here you will also need to develop a Logistic regression code that you will use to compare with the Neural Network code. The data set we propose are the so-called "Wisconsin Breat Cancer Data":"https://www.kaggle.com/uciml/breast-cancer-wisconsin-data" data set of images representing various features of tumors. These are discussed intensively in the lecture notes, see for example the slides from "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html". A longer explanation with links to the scientific literature can be found at the "Machine Learning repository of the University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29". Feel free to consult this site and the pertinent literature. You can find more information about this at the "Scikit-Learn site":"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html" or at the "University of California at Irvine":"https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)". However, if you would like to study other data sets, feel free to propose other sets. What we list here are mere suggestions from our -side. If you opt for another data set, consider using a set which -has been studied in the scientific literature. This makes it easier -for you to compare and analyze your results. Comparing with existing results from the scientific literature is also an essential -element of the scientific discussion. The University of California at Irvine with its Machine Learning repository at URL:"https://archive.ics.uci.edu/ml/index.php" is an excellent site to look up for examples and inspiration. "Kaggle.com":"https://www.kaggle.com/" is an equally interesting site. Feel free to explore these sites. +side. If you opt for another data set, consider using a set which has +been studied in the scientific literature. This makes it easier for +you to compare and analyze your results. Comparing with existing +results from the scientific literature is also an essential element of +the scientific discussion. The University of California at Irvine +with its Machine Learning repository at +URL:"https://archive.ics.uci.edu/ml/index.php" is an excellent site to +look up for examples and +inspiration. "Kaggle.com":"https://www.kaggle.com/" is an equally +interesting site. Feel free to explore these sites. We will start with a regression problem and we will reuse our codes from project 1 starting with writing our own Stochastic Gradient Descent (SGD) code. @@ -37,13 +44,26 @@ We will start with a regression problem and we will reuse our codes from project === Part a): Write your own Stochastic Gradient Descent code, first step === In order to get started, we will now replace in our standard ordinary -least squares (OLS) and Ridge regression codes (from project 1) the matrix inversion -algorithm with our own SGD code. You can choose whether you want to -add the momentum SGD optionality or other SGD variants such as RMSprop -or ADAgrad or ADAM. The lecture notes from "week 39 contain more -details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" +least squares (OLS) and Ridge regression codes (from project 1) the +matrix inversion algorithm with our own gradient descent (GD) and SGD +codes. You can use the Franke function or the terrain data from +project 1. However, we recommend using a simpler function like +$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials. +You can obviously test your final codes against for example the Franke +function. -Perform an analysis of the results for OLS and Ridge regression as + +You should include in your analysis of the GD and SGD codes the following elements +o A plain gradient descent with a fixed learning rate (you will need to tune it). +o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). +o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc) +o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD. +o Add RMSprop and Adam to your library of methods for tuning the learning rate. +The lecture notes from "week 39 contain more +details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples. + +In summary, you should +perform an analysis of the results for OLS and Ridge regression as function of the chosen learning rates, the number of mini-batches and epochs as well as algorithm for scaling the learning rate. You can also compare your own results with those that can be obtained using @@ -51,7 +71,14 @@ for example _Scikit-Learn_'s various SGD options. Discuss your results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and the learning rate $\eta$. Discuss your results. -You will need your SGD code for the setup of the Neural Network and Logistic Regression codes. You will find the Python "Seaborn package":"https://seaborn.pydata.org/generated/seaborn.heatmap.html" useful when plotting the results as function of the learning rate $\eta$ and the hyper-parameter $\lambda$ when you use Ridge regression. +You will need your SGD code for the setup of the Neural Network and +Logistic Regression codes. You will find the Python "Seaborn +package":"https://seaborn.pydata.org/generated/seaborn.heatmap.html" +useful when plotting the results as function of the learning rate +$\eta$ and the hyper-parameter $\lambda$ when you use Ridge +regression. + +We recommend reading chapter 8 on optimization from the textbook of "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning. === Part b): Writing your own Neural Network code === @@ -60,9 +87,11 @@ write your own Feed Forward Neural Network code implementing the back propagation algorithm discussed in the lecture slides from "week 40":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week40.html" and "week 41":"https://compphysics.github.io/MachineLearning/doc/pub/week41/html/week41.html". -We will focus on a regression problem first and study either the +We will focus on a regression problem first and study either the simple second-order polynomial from part a) or the Franke function or terrain data (or both or other data sets) from -project 1. Discuss again your choice of cost function. +project 1. + +Discuss again your choice of cost function. Write an FFNN code for regression with a flexible number of hidden layers and nodes using the Sigmoid function as activation function for @@ -70,12 +99,12 @@ the hidden layers. Initialize the weights using a normal distribution. How would you initialize the biases? And which activation function would you select for the final output layer? -Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1. +Train your network and compare the results with those from your OLS and Ridge Regression codes from project 1 if you use the Franke function or the terrain data. You should test your results against a similar code using _Scikit-Learn_ (see the examples in the above lecture notes from week 41) or _tensorflow/keras_. Comment your results and give a critical discussion of the results obtained with the Linear Regression code and your own Neural Network -code. Compare the results with those from project 1. +code. Make an analysis of the regularization parameters and the learning rates employed to find the optimal MSE and $R2$ scores. A useful reference on the back progagation algorithm is "Nielsen's @@ -167,7 +196,7 @@ o The textbook of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The El o "Mehta et al, arXiv 1803.08823":"https://arxiv.org/abs/1803.08823", *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823. - +o "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/", *Deep Learning*. ===== Introduction to numerical projects ===== @@ -214,39 +243,3 @@ we encourage you to collaborate. Optimal working groups consist of 2-3 students. You can then hand in a common report. - -===== Software and needed installations ===== - -If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, -we recommend that you install the following Python packages via _pip_ as -o pip install numpy scipy matplotlib ipython scikit-learn tensorflow sympy pandas pillow -For Python3, replace _pip_ with _pip3_. - -See below for a discussion of _tensorflow_ and _scikit-learn_. - -For OSX users we recommend also, after having installed Xcode, to install _brew_. Brew allows -for a seamless installation of additional software via for example -o brew install python3 - -For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution -you can use _pip_ as well and simply install Python as -o sudo apt-get install python3 (or python for python2.7) -etc etc. - -If you don't want to install various Python packages with their dependencies separately, we recommend two widely used distrubutions which set up all relevant dependencies for Python, namely -o "Anaconda":"https://docs.anaconda.com/" Anaconda is an open source distribution of the Python and R programming languages for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment. Package versions are managed by the package management system _conda_ -o "Enthought canopy":"https://www.enthought.com/product/canopy/" is a Python distribution for scientific and analytic computing distribution and analysis environment, available for free and under a commercial license. - -Popular software packages written in Python for ML are - -* "Scikit-learn":"http://scikit-learn.org/stable/", -* "Tensorflow":"https://www.tensorflow.org/", -* "PyTorch":"http://pytorch.org/" and -* "Keras":"https://keras.io/". -These are all freely available at their respective GitHub sites. They -encompass communities of developers in the thousands or more. And the number -of code developers and contributors keeps increasing. - - - -