From 984f8aa5112dd6b23aa1d1a5f2b25a0b989c3fa4 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Sat, 7 Oct 2023 22:20:29 +0200 Subject: [PATCH] Update exercisesweek41.do.txt --- doc/src/week41/exercisesweek41.do.txt | 102 +++++++++----------------- 1 file changed, 36 insertions(+), 66 deletions(-) diff --git a/doc/src/week41/exercisesweek41.do.txt b/doc/src/week41/exercisesweek41.do.txt index 5134c86e1..abb7bc2e9 100644 --- a/doc/src/week41/exercisesweek41.do.txt +++ b/doc/src/week41/exercisesweek41.do.txt @@ -1,77 +1,47 @@ -TITLE: Exercises week 38 -AUTHOR: September 18-22, 2023 -DATE: Deadline is Sunday September 24 at midnight +TITLE: Exercises week 41 +AUTHOR: October 9-13, 2023 +DATE: Deadline is Sunday October 15 at midnight ===== Overarching aims of the exercises this week ===== -The aim of the exercises this week is to derive the equations for the bias-variance tradeoff to be used in project 1 as well as testing this for a simpler function using the bootstrap method. The exercises here can be reused in project 1 as well. - -Consider a -dataset $\mathcal{L}$ consisting of the data -$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. - -We assume that the true data is generated from a noisy model - -!bt -\[ -\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}. -\] -!et - -Here $\epsilon$ is normally distributed with mean zero and standard -deviation $\sigma^2$. - -In our derivation of the ordinary least squares method we defined -an approximation to the function $f$ in terms of the parameters -$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, -that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. - -The parameters $\bm{\beta}$ are in turn found by optimizing the mean -squared error via the so-called cost function - -!bt -\[ -C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. -\] -!et -Here the expected value $\mathbb{E}$ is the sample value. - -Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a -term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise. -That is, show that -!bt -\[ -\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathrm{Bias}[\tilde{y}]+\mathrm{var}[\tilde{y}]+\sigma^2, -\] -!et -with -!bt -\[ -\mathrm{Bias}[\tilde{y}]=\mathbb{E}\left[\left(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2\right], -\] -!et -and -!bt -\[ -\mathrm{var}[\tilde{y}]=\mathbb{E}\left[\left(\tilde{\bm{y}}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2\right]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2. -\] -!et +The aim of the exercises this week - -Explain what the terms mean and discuss their interpretations. - -Perform then a bias-variance analysis of a simple one-dimensional (or other models of your choice) function by -studying the MSE value as function of the complexity of your model. Use ordinary least squares only. - -Discuss the bias and variance trade-off as function -of your model complexity (the degree of the polynomial) and the number -of data points, and possibly also your training and test data using the _bootstrap_ resampling method. -You can follow the code example in the jupyter-book at URL:"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff". +In order to get started, we will now replace in our standard ordinary +least squares (OLS) and Ridge regression codes (from project 1) the +matrix inversion algorithm with our own gradient descent (GD) and SGD +codes. You can use the Franke function or the terrain data from +project 1. However, we recommend using a simpler function like +$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials. +You can obviously test your final codes against for example the Franke +function. -See also the whiteboard notes from week 37 at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf" +You should include in your analysis of the GD and SGD codes the following elements +o A plain gradient descent with a fixed learning rate (you will need to tune it). +o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). +o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc) +o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD. +o Add RMSprop and Adam to your library of methods for tuning the learning rate. +The lecture notes from "weeks 39 and 40 contain more +details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples. +In summary, you should +perform an analysis of the results for OLS and Ridge regression as +function of the chosen learning rates, the number of mini-batches and +epochs as well as algorithm for scaling the learning rate. You can +also compare your own results with those that can be obtained using +for example _Scikit-Learn_'s various SGD options. Discuss your +results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and +the learning rate $\eta$. Discuss your results. +You will need your SGD code for the setup of the Neural Network and +Logistic Regression codes. You will find the Python "Seaborn +package":"https://seaborn.pydata.org/generated/seaborn.heatmap.html" +useful when plotting the results as function of the learning rate +$\eta$ and the hyper-parameter $\lambda$ when you use Ridge +regression. + +We recommend reading chapter 8 on optimization from the textbook of "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning.