lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update exercisesweek41.do.txt

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-07 22:20:29 +02:00
parent 1081b85378
commit 984f8aa511
1 changed files with 36 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week41/exercisesweek41.do.txt
+36 -66
View File
@@ -1,77 +1,47 @@
TITLE: Exercises week 38
AUTHOR: September 18-22, 2023
DATE: Deadline is Sunday September 24 at midnight
TITLE: Exercises week 41
AUTHOR: October 9-13, 2023
DATE: Deadline is Sunday October 15 at midnight
===== Overarching aims of the exercises this week =====
The aim of the exercises this week is to derive the equations for the bias-variance tradeoff to be used in project 1 as well as testing this for a simpler function using the bootstrap method. The exercises here can be reused in project 1 as well.
Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
We assume that the true data is generated from a noisy model
!bt
\[
\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}.
\]
!et
Here $\epsilon$ is normally distributed with mean zero and standard
deviation $\sigma^2$.
In our derivation of the ordinary least squares method we defined
an approximation to the function $f$ in terms of the parameters
$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model,
that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$.
The parameters $\bm{\beta}$ are in turn found by optimizing the mean
squared error via the so-called cost function
!bt
\[
C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
\]
!et
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
!bt
\[
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathrm{Bias}[\tilde{y}]+\mathrm{var}[\tilde{y}]+\sigma^2,
\]
!et
with
!bt
\[
\mathrm{Bias}[\tilde{y}]=\mathbb{E}\left[\left(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2\right],
\]
!et
and
!bt
\[
\mathrm{var}[\tilde{y}]=\mathbb{E}\left[\left(\tilde{\bm{y}}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2\right]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2.
\]
!et
The aim of the exercises this week
Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of a simple one-dimensional (or other models of your choice) function by
studying the MSE value as function of the complexity of your model. Use ordinary least squares only.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the _bootstrap_ resampling method.
You can follow the code example in the jupyter-book at URL:"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff".
In order to get started, we will now replace in our standard ordinary
least squares (OLS) and Ridge regression codes (from project 1) the
matrix inversion algorithm with our own gradient descent (GD) and SGD
codes. You can use the Franke function or the terrain data from
project 1. However, we recommend using a simpler function like
$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.
You can obviously test your final codes against for example the Franke
function.
See also the whiteboard notes from week 37 at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf"
You should include in your analysis of the GD and SGD codes the following elements
o A plain gradient descent with a fixed learning rate (you will need to tune it).
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
o Add RMSprop and Adam to your library of methods for tuning the learning rate.
The lecture notes from "weeks 39 and 40 contain more
details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples.
In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
epochs as well as algorithm for scaling the learning rate. You can
also compare your own results with those that can be obtained using
for example _Scikit-Learn_'s various SGD options. Discuss your
results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\lambda$ and
the learning rate $\eta$. Discuss your results.
You will need your SGD code for the setup of the Neural Network and
Logistic Regression codes. You will find the Python "Seaborn
package":"https://seaborn.pydata.org/generated/seaborn.heatmap.html"
useful when plotting the results as function of the learning rate
$\eta$ and the hyper-parameter $\lambda$ when you use Ridge
regression.
We recommend reading chapter 8 on optimization from the textbook of "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning.