lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-09-03 21:02:06 +02:00
parent 530f5a235b
commit a273b77783
10 changed files with 4466 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/Projects/2023/Project1/Project1.do.txt
+64 -66
View File
@@ -121,64 +121,7 @@ plt.show()
!ec
=== Part a): Paper and pencil part (also as weekly exercise for week 36) ===
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570").
The assumption we have made is
that there exists a continuous function $f(\bm{x})$ and a normal distributed error $\bm{\varepsilon}\sim N(0, \sigma^2)$
which describes our data
!bt
\[
\bm{y} = f(\bm{x})+\bm{\varepsilon}
\]
!et
We then approximate this function $f(\bm{x})$ with our model $\bm{\tilde{y}}$ from the solution of the linear regression equations (ordinary least squares OLS), that is our
function $f$ is approximated by $\bm{\tilde{y}}$ where we minimized $(\bm{y}-\bm{\tilde{y}})^2$, with
!bt
\[
\bm{\tilde{y}} = \bm{X}\bm{\beta}.
\]
!et
The matrix $\bm{X}$ is the so-called design or feature matrix.
Show that the expectation value of $\bm{y}$ for a given element $i$
!bt
\[
\mathbb{E}(y_i) =\sum_{j}x_{ij} \beta_j=\mathbf{X}_{i, \ast} \, \bm{\beta},
\]
!et
and that
its variance is
!bt
\[
\mbox{Var}(y_i) = \sigma^2.
\]
!et
Hence, $y_i \sim N( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$.
With the OLS expressions for the optimal parameters $\bm{\hat{\beta}}$ show that
!bt
\[
\mathbb{E}(\bm{\hat{\beta}}) = \bm{\beta}.
\]
!et
Show finally that the variance of $\bm{\beta}$ is
!bt
\[
\mbox{Var}(\bm{\hat{\beta}}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
\]
!et
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$. .
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
=== Part b) : Ordinary Least Square (OLS) on the Franke function ===
=== Part a) : Ordinary Least Square (OLS) on the Franke function ===
We will generate our own dataset for a function
$\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function
@@ -241,20 +184,16 @@ You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
=== Part c): Ridge Regression on the Franke function ===
=== Part b): Adding Ridge and Lasso Regression on the Franke function ===
Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
exercise. Perform the same bootstrap analysis as in the
part c) (for the same polynomials) and the cross-validation in part d) but now for different values of $\lambda$. Compare and
exercise.
Perform the same analysis as you did in the previous exercise but now for different values of $\lambda$. Compare and
analyze your results with those obtained in parts b-d). Study the
dependence on $\lambda$.
Study also the bias-variance trade-off as function of various values of
the parameter $\lambda$. For the bias-variance trade-off, use the _bootstrap_ resampling method. Comment your results.
=== Part f): Lasso Regression on the Franke function with resampling ===
This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
you can also use the functionalities of _Scikit-Learn_ (recommended).
@@ -263,6 +202,65 @@ critical discussion of the three methods and a judgement of which
model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the _bootstrap_ resampling technique and an analysis of the mean squared error using cross-validation.
=== Part a): Paper and pencil part ===
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570").
The assumption we have made is
that there exists a continuous function $f(\bm{x})$ and a normal distributed error $\bm{\varepsilon}\sim N(0, \sigma^2)$
which describes our data
!bt
\[
\bm{y} = f(\bm{x})+\bm{\varepsilon}
\]
!et
We then approximate this function $f(\bm{x})$ with our model $\bm{\tilde{y}}$ from the solution of the linear regression equations (ordinary least squares OLS), that is our
function $f$ is approximated by $\bm{\tilde{y}}$ where we minimized $(\bm{y}-\bm{\tilde{y}})^2$, with
!bt
\[
\bm{\tilde{y}} = \bm{X}\bm{\beta}.
\]
!et
The matrix $\bm{X}$ is the so-called design or feature matrix.
Show that the expectation value of $\bm{y}$ for a given element $i$
!bt
\[
\mathbb{E}(y_i) =\sum_{j}x_{ij} \beta_j=\mathbf{X}_{i, \ast} \, \bm{\beta},
\]
!et
and that
its variance is
!bt
\[
\mbox{Var}(y_i) = \sigma^2.
\]
!et
Hence, $y_i \sim N( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$.
With the OLS expressions for the optimal parameters $\bm{\hat{\beta}}$ show that
!bt
\[
\mathbb{E}(\bm{\hat{\beta}}) = \bm{\beta}.
\]
!et
Show finally that the variance of $\bm{\beta}$ is
!bt
\[
\mbox{Var}(\bm{\hat{\beta}}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
\]
!et
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$. .
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
=== Part c): Bias-variance trade-off and resampling techniques ===
doc/src/Projects/2023/Project1/make.sh
+1 -1
View File
@@ -51,7 +51,7 @@ mv -f $name.pdf ${name}.pdf
cp $name.tex ${name}.tex
# Publish
dest=../../../../Projects/2022
dest=../../../../Projects/2023
if [ ! -d $dest/$name ]; then
mkdir $dest/$name
mkdir $dest/$name/pdf