lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

project 1 final version(?)

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-09-04 08:04:42 +02:00
parent a273b77783
commit 2a445d0f02
9 changed files with 303 additions and 220 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2023/Project1/html/Project1.html
+41 -28
View File
@@ -126,24 +126,27 @@ div.toc p,a {
3,
None,
'part-a-ordinary-least-square-ols-on-the-franke-function'),
('Part b): Adding Ridge and Lasso Regression on the Franke '
'function',
('Part b): Adding Ridge regression for the Franke function',
3,
None,
'part-b-adding-ridge-and-lasso-regression-on-the-franke-function'),
('Part a): Paper and pencil part',
'part-b-adding-ridge-regression-for-the-franke-function'),
('Part c): Adding Lasso for the Franke function',
3,
None,
'part-a-paper-and-pencil-part'),
('Part c): Bias-variance trade-off and resampling techniques',
'part-c-adding-lasso-for-the-franke-function'),
('Part d): Paper and pencil part',
3,
None,
'part-c-bias-variance-trade-off-and-resampling-techniques'),
('Part d): Cross-validation as resampling techniques, adding '
'part-d-paper-and-pencil-part'),
('Part e): Bias-variance trade-off and resampling techniques',
3,
None,
'part-e-bias-variance-trade-off-and-resampling-techniques'),
('Part f): Cross-validation as resampling techniques, adding '
'more complexity',
3,
None,
'part-d-cross-validation-as-resampling-techniques-adding-more-complexity'),
'part-f-cross-validation-as-resampling-techniques-adding-more-complexity'),
('Part g): Analysis of real data',
3,
None,
@@ -210,14 +213,14 @@ for example
we detail how to write a report. Furthermore, at
<a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/</tt></a>
you can find examples of previous reports. How to write reports will
also be discussed during lectures and at the various lab sessions.
also be discussed during the various lab sessions.
</p>
<p><b>A small recommendation when developing the codes here</b>. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
of week 35. A simple test, as discussed during the lectures the first
of weeks 35 and 36. A simple test, as discussed during the lectures the first
two weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
@@ -249,12 +252,17 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}
\end{align*}
$$
<p>The function will be defined for \( x,y\in [0,1] \). Our first step will
<p>The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.</p>
<p>Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y,
x^2, y^2, xy, \dots] \). We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
technique. As discussed in the lectures for weeks 35 and 36,, we can
technique.
</p>
<p>We can
use a uniform distribution to set up the arrays of values for \( x \) and
\( y \), or as in the example below just a set of fixed values for \( x \) and
\( y \) with a given step size. We will fit a function (for example a
@@ -389,26 +397,29 @@ approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
<p>You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
</p>
<h3 id="part-b-adding-ridge-and-lasso-regression-on-the-franke-function">Part b): Adding Ridge and Lasso Regression on the Franke function </h3>
<h3 id="part-b-adding-ridge-regression-for-the-franke-function">Part b): Adding Ridge regression for the Franke function </h3>
<p>Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
exercise.
exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
</p>
<p>Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and
analyze your results with those obtained in parts b-d). Study the
analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on \( \lambda \).
</p>
<h3 id="part-c-adding-lasso-for-the-franke-function">Part c): Adding Lasso for the Franke function </h3>
<p>This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
you can also use the functionalities of <b>Scikit-Learn</b> (recommended).
you can also use the functionalities of <b>Scikit-Learn</b> (recommended). Keep in mind that the library <b>Scikit-Learn</b> excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the <b>bootstrap</b> resampling technique and an analysis of the mean squared error using cross-validation.
model fits the data best.
</p>
<h3 id="part-a-paper-and-pencil-part">Part a): Paper and pencil part </h3>
<p>v</p>
<h3 id="part-d-paper-and-pencil-part">Part d): Paper and pencil part </h3>
<p>This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of <a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>).</p>
@@ -458,9 +469,11 @@ $$
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). .
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
</p>
<h3 id="part-c-bias-variance-trade-off-and-resampling-techniques">Part c): Bias-variance trade-off and resampling techniques </h3>
<h3 id="part-e-bias-variance-trade-off-and-resampling-techniques">Part e): Bias-variance trade-off and resampling techniques </h3>
<p>Our aim here is to study the bias-variance trade-off by implementing the <b>bootstrap</b> resampling technique.</p>
<p>Our aim here is to study the bias-variance trade-off by implementing the <b>bootstrap</b> resampling technique.
<b>We will only use the simpler ordinary least squares here</b>.
</p>
<p>With a code which does OLS and includes resampling techniques,
we will now discuss the bias-variance trade-off in the context of
@@ -483,7 +496,7 @@ dataset \( \mathcal{L} \) consisting of the data
\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
</p>
<p>As in part a), we assume that the true data is generated from a noisy model</p>
<p>As in part d), we assume that the true data is generated from a noisy model</p>
$$
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
@@ -540,12 +553,10 @@ of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
You can follow the code example in the jupyter-book at <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff</tt></a>.
</p>
<h3 id="part-d-cross-validation-as-resampling-techniques-adding-more-complexity">Part d): Cross-validation as resampling techniques, adding more complexity </h3>
<h3 id="part-f-cross-validation-as-resampling-techniques-adding-more-complexity">Part f): Cross-validation as resampling techniques, adding more complexity </h3>
<p>The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data if you think this is needed.
resampling technique, the so-called cross-validation method.
</p>
<p>Implement the \( k \)-fold cross-validation algorithm (write your own
@@ -559,17 +570,19 @@ you got from your <b>bootstrap</b> code. Comment your results. Try \( 5-10 \)
folds. You can also compare your own cross-validation code with the
one provided by <b>Scikit-Learn</b>.
</p>
<p>In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression. </p>
<h3 id="part-g-analysis-of-real-data">Part g): Analysis of real data </h3>
<p>With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
essentially repeat in this exercise what was done in exercises 1-5. However, we
essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
<a href="https://earthexplorer.usgs.gov/" target="_blank"><tt>https://earthexplorer.usgs.gov/</tt></a>,
</p>
<p>Or, if you prefer, we have placed selected datafiles at <a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles</tt></a></p>
<p>Or, if you prefer, we have placed selected datafiles at <a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles</tt></a></p>
<p>In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want