update week 35

doc/src/week35/exercisesweek35.do.txt

@@ -0,0 +1,151 @@
+TITLE: Exercises week 35
+AUTHOR: FYS-STK3155/4155
+DATE: August 28-September 1, 2023
+
+
+
+
+
+===== Exercise: making your own data and exploring scikit-learn =====
+
+
+We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$.
+The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
+!bc pycod
+x = np.random.rand(100,1)
+y = 2.0+5*x*x+0.1*np.random.randn(100,1)
+!ec
+
+o Write your own code (following the examples under the "regression notes":"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter1.html") for computing the parametrization of the data set fitting a second-order polynomial. 
+o Use thereafter _scikit-learn_ (see again the examples in the regression slides) and compare with your own code.   
+o Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
+!bt 
+\[ MSE(\bm{y},\bm{\tilde{y}}) = \frac{1}{n}
+\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, 
+\] 
+!et 
+and the $R^2$ score function.
+If $\tilde{\bm{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
+!bt
+\[
+R^2(\bm{y}, \tilde{\bm{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+\]
+!et
+where we have defined the mean value  of $\bm{y}$ as
+!bt
+\[
+\bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+\]
+!et
+You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions. 
+Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.
+
+===== Exercise: Split data in test and training data =====
+
+In this exercise we want you to to compute the MSE for the training
+data and the test data as function of the complexity of a polynomial,
+that is the degree of a given polynomial.
+
+The aim is to reproduce Figure 2.11 of "Hastie et al":"https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf".
+
+Our data is defined by $x\in [-3,3]$ with a total of for example $n=100$ data points. You should try to vary the number of data points $n$ in your analysis.
+!bc pycod
+np.random.seed()
+n = 100
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+!ec
+where $y$ is the function we want to fit with a given polynomial.
+!bsubex
+Write a first code which sets up a design matrix $X$ defined by a fifth-order polynomial and split your data set in training and test data. 
+!esubex
+
+!bsubex
+Write thereafter (using either _scikit-learn_ or your matrix inversion code using for example _numpy_)
+and perform an ordinary least squares fitting and compute the mean squared error for the training data and the test data. These calculations should apply to a model given by a fifth-order polynomial.
+!esubex
+
+!bsubex
+Add now a model which allows you to make polynomials up to degree $15$.  Perform a standard OLS fitting of the training data and compute the MSE for the training and test data and plot both test and training data MSE as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?
+!esubex
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+===== Exercise: Analytical exercises  =====
+
+In this exercise we derive the expressions for various derivatives of
+products of vectors and matrices. Such derivatives are central to the
+optimization of various cost functions. Although we will often use
+automatic differentiation in actual calculations, to be able to have
+analytical expressions is extremely helpful in case we have simpler
+derivatives as well as when we analyze various properties (like second
+derivatives) of the chosen cost functions.  Vectors are always written
+as boldfaced lower case letters and matrices as upper case boldfaced
+letters.
+
+Show that
+!bt
+\[
+\frac{\partial (\bm{b}^T\bm{a})}{\partial \bm{a}} = \bm{b},
+\]
+!et
+and
+!bt
+\[
+\frac{\partial (\bm{a}^T\bm{A}\bm{a})}{\partial \bm{a}} = \bm{a}^T(\bm{A}+\bm{A}^T),
+\]
+!et
+and
+!bt
+\[
+\frac{\partial \left(\bm{x}-\bm{A}\bm{s}\right)^T\left(\bm{x}-\bm{A}\bm{s}\right)}{\partial \bm{s}} = -2\left(\bm{x}-\bm{A}\bm{s}\right)^T\bm{A},
+\]
+!et
+and finally find the second derivative of this function with respect to the vector $\bm{s}$.
+
+_Hint_: In these exercises it is always useful to write out with summation indices the various quantities.
+As an example, consider the function
+
+!bt
+\[
+f(\bm{x}) =\bm{A}\bm{x},
+\]
+!et
+which reads for a specific component $f_i$ (we define the matrix $\bm{A}$ to have dimension $n\times n$ and the vector $\bm{x} to have length $n$)
+
+!bt
+\[
+f_i =\sum_{j=0}^{n-1}a_{ij}x_j, 
+\]
+!et
+which leads to
+!bt
+\[
+\frac{\partial f_i}{\partial x_j}= a_{ij},
+\]
+!et
+and written out in terms of the vector $\bm{x}$ we have
+!bt
+\[
+\frac{\partial f(\bm{x})}{\partial \bm{x}}= \bm{A}.
+\]
+!et
+
+