lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update p1

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-01 12:21:53 +02:00
parent da97fdfc3f
commit dccb74ba38
22 changed files with 1858 additions and 6798 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2025/Project1/html/Project1.html
+117 -329
View File
@@ -8,8 +8,8 @@ doconce format html Project1.do.txt --pygments_html_style=default --html_style=b
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Project 1 on Machine Learning, deadline October 7 (midnight), 2024">
<title>Project 1 on Machine Learning, deadline October 7 (midnight), 2024</title>
<meta name="description" content="Project 1 on Machine Learning, deadline October 6 (midnight), 2025">
<title>Project 1 on Machine Learning, deadline October 6 (midnight), 2025</title>
<style type="text/css">
/* bloodish style */
body {
@@ -123,39 +123,40 @@ div.toc p,a {
2,
None,
'regression-analysis-and-resampling-methods'),
('Description of two-dimensional function',
('Part a : Ordinary Least Square (OLS) for the Runge function',
3,
None,
'description-of-two-dimensional-function'),
('Part a) : Ordinary Least Square (OLS) on the Franke function',
'part-a-ordinary-least-square-ols-for-the-runge-function'),
('Part b: Adding Ridge regression for the Runge function',
3,
None,
'part-a-ordinary-least-square-ols-on-the-franke-function'),
('Part b): Adding Ridge regression for the Franke function',
'part-b-adding-ridge-regression-for-the-runge-function'),
('Part c: Writing your own gradient descent code',
3,
None,
'part-b-adding-ridge-regression-for-the-franke-function'),
('Part c): Adding Lasso for the Franke function',
'part-c-writing-your-own-gradient-descent-code'),
('Part d: Including momentum and more advanced ways to update '
'the learning the rate',
3,
None,
'part-c-adding-lasso-for-the-franke-function'),
('Part d): Paper and pencil part',
'part-d-including-momentum-and-more-advanced-ways-to-update-the-learning-the-rate'),
('Part e: Writing our own code for Lasso regression',
3,
None,
'part-d-paper-and-pencil-part'),
('Part e): Bias-variance trade-off and resampling techniques',
'part-e-writing-our-own-code-for-lasso-regression'),
('Part f: Stochastic gradient descent',
3,
None,
'part-e-bias-variance-trade-off-and-resampling-techniques'),
('Part f): Cross-validation as resampling techniques, adding '
'part-f-stochastic-gradient-descent'),
('Part g: Bias-variance trade-off and resampling techniques',
3,
None,
'part-g-bias-variance-trade-off-and-resampling-techniques'),
('Part h): Cross-validation as resampling techniques, adding '
'more complexity',
3,
None,
'part-f-cross-validation-as-resampling-techniques-adding-more-complexity'),
('Part g): Analysis of real data',
3,
None,
'part-g-analysis-of-real-data'),
'part-h-cross-validation-as-resampling-techniques-adding-more-complexity'),
('Background literature', 2, None, 'background-literature'),
('Introduction to numerical projects',
2,
@@ -190,12 +191,12 @@ MathJax.Hub.Config({
<!-- ------------------- main content ---------------------- -->
<center>
<h1>Project 1 on Machine Learning, deadline October 7 (midnight), 2024</h1>
<h1>Project 1 on Machine Learning, deadline October 6 (midnight), 2025</h1>
</center> <!-- document title -->
<!-- author(s): <a href="http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" target="_blank">Data Analysis and Machine Learning FYS-STK3155/FYS4155</a> -->
<!-- author(s): Data Analysis and Machine Learning FYS-STK3155/FYS4155 -->
<center>
<b><a href="http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" target="_blank">Data Analysis and Machine Learning FYS-STK3155/FYS4155</a></b>
<b>Data Analysis and Machine Learning FYS-STK3155/FYS4155</b>
</center>
<!-- institution -->
<center>
@@ -209,24 +210,21 @@ MathJax.Hub.Config({
<h2 id="preamble-note-on-writing-reports-using-reference-material-ai-and-other-tools">Preamble: Note on writing reports, using reference material, AI and other tools </h2>
<p>We want you to answer the three different projects by handing in
reports written like a standard scientific/technical report. The link
at
<a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ProjectWriting/projectwriting.ipynb" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ProjectWriting/projectwriting.ipynb</tt></a>
gives some guidance. See also the grading suggestion at
<a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md</tt></a>.
</p>
<p>Furthermore, at
<a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/</tt></a>
reports written like a standard scientific/technical report. The links
at <a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects</tt></a>
Furthermore, at the same link,
you can find examples of previous reports. How to write reports will
also be discussed during the various lab sessions. Please do ask us if you are in doubt.
</p>
<p>When using codes and material from other sources, you should refer to these in the bibliography of your report, indicating wherefrom you for example
got the code, whether this is from the lecture notes, softwares like
Scikit-Learn, TensorFlow, PyTorch or other sources. These should
always be cited correctly. How to cite some of the libraries is often
indicated from their corresponding GitHub sites or websites, see for example how to cite Scikit-Learn at <a href="https://scikit-learn.org/dev/about.html" target="_blank"><tt>https://scikit-learn.org/dev/about.html</tt></a>.
<p>When using codes and material from other sources, you should refer to
these in the bibliography of your report, indicating wherefrom you for
example got the code, whether this is from the lecture notes,
softwares like Scikit-Learn, TensorFlow, PyTorch or other sources such
AI software. These should always be cited correctly. How to cite some
of the libraries is often indicated from their corresponding GitHub
sites or websites, see for example how to cite Scikit-Learn at
<a href="https://scikit-learn.org/dev/about.html" target="_blank"><tt>https://scikit-learn.org/dev/about.html</tt></a>.
</p>
<p>We enocurage you to use tools like
@@ -234,6 +232,24 @@ indicated from their corresponding GitHub sites or websites, see for example how
please do cite it properly and include (if possible) your questions and answers as an addition to the report. This can
be uploaded to for example your website, GitHub/GitLab or similar as supplemental material.
</p>
<p>If you would like to study other data sets, feel free to propose other
sets. What we have proposed here are mere suggestions from our
side. If you opt for another data set, consider using a set which has
been studied in the scientific literature. This makes it easier for
you to compare and analyze your results. Comparing with existing
results from the scientific literature is also an essential element of
the scientific discussion. The University of California at Irvine
with its Machine Learning repository at
<a href="https://archive.ics.uci.edu/ml/index.php" target="_blank"><tt>https://archive.ics.uci.edu/ml/index.php</tt></a> is an excellent site to
look up for examples and
inspiration. <a href="https://www.kaggle.com/" target="_blank">Kaggle.com</a> is an equally
interesting site. Feel free to explore these sites. When selecting
other data sets, make sure these are sets used for regression problems
(not classification).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="regression-analysis-and-resampling-methods">Regression analysis and resampling methods </h2>
<p>The main aim of this project is to study in more detail various
@@ -242,141 +258,38 @@ In addition to the scientific part, in this course we want also to
give you an experience in writing scientific reports.
</p>
<p><b>A small recommendation when developing the codes here</b>. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
of weeks 35 and 36. A simple test, as discussed during the lectures the first
three weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
the one-dimensional test function with the two-dimensional <b>Franke</b> function
discussed here.
<p>We will study how to fit polynomials to specific
one-dimensional functions (feel free to replace the suggested function with more complicated ones).
</p>
<p>The Franke function serves as a stepping stone towards the analysis of
real topographic data. The latter is the last part of this project.
</p>
<h3 id="description-of-two-dimensional-function">Description of two-dimensional function </h3>
<p>We will first study how to fit polynomials to a specific
two-dimensional function called <a href="http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf" target="_blank">Franke's
function</a>. This
is a function which has been widely used when testing various
interpolation and fitting algorithms. Furthermore, after having
established the model and the method, we will employ resamling
techniques such as cross-validation and/or bootstrap in order to perform a
proper assessment of our models. We will also study in detail the
so-called Bias-Variance trade off.
</p>
<p>The Franke function, which is a weighted sum of four exponentials reads as follows</p>
<p>We will use Runge's function (see <a href="https://en.wikipedia.org/wiki/Runge%27s_phenomenon" target="_blank"><tt>https://en.wikipedia.org/wiki/Runge%27s_phenomenon</tt></a> for a discussion). The one-dimensional function we will study is</p>
$$
\begin{align*}
f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
\end{align*}
f(x) = \frac{1}{1+25x^2}.
$$
<p>The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.</p>
<p>Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an \( x \) and a \( y \) dependence of the form \( [x, y,
x^2, y^2, xy, \dots] \). We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
technique.
</p>
<p>We can
use a uniform distribution to set up the arrays of values for \( x \) and
\( y \), or as in the example below just a set of fixed values for \( x \) and
\( y \) with a given step size. We will fit a function (for example a
polynomial) of \( x \) and \( y \). Thereafter we will repeat much of the
<p>Our first step will be to perform an OLS regression analysis of this
function, trying out a polynomial fit with an \( x \) dependence of the
form \( [x,x^2,\dots] \). We can use a uniform distribution to set up the
arrays of values for \( x \in [-1,1] \), or alternatively use a fixed step size.
Thereafter we will repeat much of the
same procedure using the Ridge and Lasso regression methods,
introducing thus a dependence on the bias (penalty) \( \lambda \).
introducing thus a dependence on the hyperparameter (penalty) \( \lambda \).
</p>
<p>Finally we are going to use (real) digital terrain data and try to
reproduce these data using the same methods. We will also try to go
beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
<p>We will also include bootstrap as a resampling technique in order to
study the so-called <b>bias-variance tradeoff</b>. After that we will
include the cross-validation technique.
</p>
<p>The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> Axes3D
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">matplotlib</span> <span style="color: #008000; font-weight: bold">import</span> cm
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">matplotlib.ticker</span> <span style="color: #008000; font-weight: bold">import</span> LinearLocator, FormatStrFormatter
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(projection <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;3d&#39;</span>)
<span style="color: #408080; font-style: italic"># Make data.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0.05</span>)
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0.05</span>)
x, y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(x,y)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">FrankeFunction</span>(x,y):
term1 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">0.25*</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>) <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>))
term2 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>((<span style="color: #666666">9*</span>x<span style="color: #666666">+1</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">/49.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.1*</span>(<span style="color: #666666">9*</span>y<span style="color: #666666">+1</span>))
term3 <span style="color: #666666">=</span> <span style="color: #666666">0.5*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-7</span>)<span style="color: #666666">**2/4.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-3</span>)<span style="color: #666666">**2</span>))
term4 <span style="color: #666666">=</span> <span style="color: #666666">-0.2*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-4</span>)<span style="color: #666666">**2</span> <span style="color: #666666">-</span> (<span style="color: #666666">9*</span>y<span style="color: #666666">-7</span>)<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">return</span> term1 <span style="color: #666666">+</span> term2 <span style="color: #666666">+</span> term3 <span style="color: #666666">+</span> term4
z <span style="color: #666666">=</span> FrankeFunction(x, y)
<span style="color: #408080; font-style: italic"># Plot the surface.</span>
surf <span style="color: #666666">=</span> ax<span style="color: #666666">.</span>plot_surface(x, y, z, cmap<span style="color: #666666">=</span>cm<span style="color: #666666">.</span>coolwarm,
linewidth<span style="color: #666666">=0</span>, antialiased<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>)
<span style="color: #408080; font-style: italic"># Customize the z axis.</span>
ax<span style="color: #666666">.</span>set_zlim(<span style="color: #666666">-0.10</span>, <span style="color: #666666">1.40</span>)
ax<span style="color: #666666">.</span>zaxis<span style="color: #666666">.</span>set_major_locator(LinearLocator(<span style="color: #666666">10</span>))
ax<span style="color: #666666">.</span>zaxis<span style="color: #666666">.</span>set_major_formatter(FormatStrFormatter(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">%.02f</span><span style="color: #BA2121">&#39;</span>))
<span style="color: #408080; font-style: italic"># Add a color bar which maps values to colors.</span>
fig<span style="color: #666666">.</span>colorbar(surf, shrink<span style="color: #666666">=0.5</span>, aspect<span style="color: #666666">=5</span>)
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>If you wish to compare your results with other on the Franke function or other popular functions tested with linear regression, see the list in Figure 1 of the article by Cook et al at <a href="https://arxiv.org/abs/2401.11694" target="_blank"><tt>https://arxiv.org/abs/2401.11694</tt></a>.</p>
<h3 id="part-a-ordinary-least-square-ols-on-the-franke-function">Part a) : Ordinary Least Square (OLS) on the Franke function </h3>
<h3 id="part-a-ordinary-least-square-ols-for-the-runge-function">Part a : Ordinary Least Square (OLS) for the Runge function </h3>
<p>We will generate our own dataset for a function
\( \mathrm{FrankeFunction}(x,y) \) with \( x,y \in [0,1] \). The function
\( f(x,y) \) is the Franke function. You should explore also the addition
\( \mathrm{Runge}(x) \) function with \( x\in [-1,1] \). You should explore also the addition
of an added stochastic noise to this function using the normal
distribution \( N(0,1) \).
</p>
<p><em>Write your own code</em> (using either a matrix inversion or a singular
value decomposition from e.g., <b>numpy</b> ) and perform a standard <b>ordinary least square regression</b>
analysis using polynomials in \( x \) and \( y \) up to fifth order.
<p><em>Write your own code</em> (using for example the pseudoinverse function <b>pinv</b> from <b>Numpy</b> ) and perform a standard <b>ordinary least square regression</b>
analysis using polynomials in \( x \) up to order \( 15 \). Explore the dependence on the number of data points and the polynomial degree.
</p>
<p>Evaluate the mean Squared error (MSE)</p>
@@ -400,8 +313,8 @@ $$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
<p>Plot the resulting scores (MSE and R$^2$) as functions of the polynomial degree (here up to polymial degree five).
Plot also the parameters \( \beta \) as you increase the order of the polynomial. Comment your results.
<p>Plot the resulting scores (MSE and R$^2$) as functions of the polynomial degree (here up to polymial degree 20).
Plot also the parameters \( \theta \) as you increase the order of the polynomial. Comment your results.
</p>
<p>Your code has to include a scaling/centering of the data (for example by
@@ -421,82 +334,62 @@ data and say test data. An accepted rule of thumb is to use
approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
</p>
<p>You can easily reuse the solutions to your exercises from week 35 and week 36.
<p>You can easily reuse the solutions to your exercises from week 35.
See also the lecture slides from week 35 and week 36.
</p>
<p>On scaling, we recommend reading the following section from the scikit-learn software description, see <a href="https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#plot-all-scaling-standard-scaler-section" target="_blank"><tt>https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#plot-all-scaling-standard-scaler-section</tt></a>.</p>
<h3 id="part-b-adding-ridge-regression-for-the-franke-function">Part b): Adding Ridge regression for the Franke function </h3>
<h3 id="part-b-adding-ridge-regression-for-the-runge-function">Part b: Adding Ridge regression for the Runge function </h3>
<p>Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
<p>Write your own code for the Ridge method as done in the previous
exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the exercise from week 36 is something you can reuse here.
</p>
<p>Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and
analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on \( \lambda \).
</p>
<h3 id="part-c-adding-lasso-for-the-franke-function">Part c): Adding Lasso for the Franke function </h3>
<h3 id="part-c-writing-your-own-gradient-descent-code">Part c: Writing your own gradient descent code </h3>
<p>This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
you can also use the functionalities of <b>Scikit-Learn</b> (recommended). Keep in mind that the library <b>Scikit-Learn</b> excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
model fits the data best.
</p>
<h3 id="part-d-paper-and-pencil-part">Part d): Paper and pencil part </h3>
<p>This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of <a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>). The exercise is also part of the weekly exercises for week 37.</p>
<p>The assumption we have made is
that there exists a continuous function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim N(0, \sigma^2) \)
which describes our data
</p>
$$
\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon}
$$
<p>We then approximate this function \( f(\boldsymbol{x}) \) with our model \( \boldsymbol{\tilde{y}} \) from the solution of the linear regression equations (ordinary least squares OLS), that is our
function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we minimized \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), with
</p>
$$
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
$$
<p>The matrix \( \boldsymbol{X} \) is the so-called design or feature matrix. </p>
<p>Show that the expectation value of \( \boldsymbol{y} \) for a given element \( i \) </p>
$$
\mathbb{E}(y_i) =\sum_{j}x_{ij} \beta_j=\mathbf{X}_{i, \ast} \, \boldsymbol{\beta},
$$
<p>and that
its variance is
</p>
$$
\mbox{Var}(y_i) = \sigma^2.
$$
<p>Hence, \( y_i \sim N( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \).
<p>Replace now the analytical expressions for the optimal parameters
\( \boldsymbol{\theta} \) with your own gradient descent code. In this exercise we
focus only on the simplest gradient descent approach with a fixed
learning rate (see the exercises from week 37 and the lecture notes
from week 36).
</p>
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\beta}} \) show that</p>
$$
\mathbb{E}(\boldsymbol{\hat{\beta}}) = \boldsymbol{\beta}.
$$
<p>Show finally that the variance of \( \boldsymbol{\beta} \) is</p>
$$
\mbox{Var}(\boldsymbol{\hat{\beta}}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
$$
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \).
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
<p>Study and compare your results from parts a) and b) with your gradient
descent approch. Discuss in particular the role of the learning rate.
</p>
<h3 id="part-e-bias-variance-trade-off-and-resampling-techniques">Part e): Bias-variance trade-off and resampling techniques </h3>
<h3 id="part-d-including-momentum-and-more-advanced-ways-to-update-the-learning-the-rate">Part d: Including momentum and more advanced ways to update the learning the rate </h3>
<p>We keep our focus on OLS and Ridge regression and update our code for
the gradient descent method by including <b>momentum</b>, <b>ADAgrad</b>,
<b>RMSprop</b> and <b>ADAM</b> as methods fro iteratively updating your learning
rate. Discuss the results and compare the different methods applied to
the one-dimensional Runge function.
</p>
<h3 id="part-e-writing-our-own-code-for-lasso-regression">Part e: Writing our own code for Lasso regression </h3>
<p>LASSO regression (see lecture slides from week 36 and week 37)
represents our first encounter with a machine learning method which
cannot be solved through analytical expressions. Use the gradient
descent methods you developed in parts c) and d) to solve the LASSO
optimization problem. You can compare your results using
the functionalities of <b>Scikit-Learn</b>.
</p>
<p>Discuss (critically) your results for the Runge function from OLS,
Ridge and LASSO regression using the various gradient descent
approaches.
</p>
<h3 id="part-f-stochastic-gradient-descent">Part f: Stochastic gradient descent </h3>
<p>Our last gradient step is to include stochastic gradient descent using the
same methods to update the learning rates as in parts c-e).
Compare and discuss your results with and without stochastic gradient and give a critical assessment of the various methods.
</p>
<h3 id="part-g-bias-variance-trade-off-and-resampling-techniques">Part g: Bias-variance trade-off and resampling techniques </h3>
<p>Our aim here is to study the bias-variance trade-off by implementing the <b>bootstrap</b> resampling technique.
<b>We will only use the simpler ordinary least squares here</b>.
@@ -571,7 +464,7 @@ $$
Explain what the terms mean and discuss their interpretations.
</p>
<p>Perform then a bias-variance analysis of the Franke function by
<p>Perform then a bias-variance analysis of the Runge function by
studying the MSE value as function of the complexity of your model.
</p>
@@ -580,14 +473,13 @@ of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
You can follow the code example in the jupyter-book at <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff</tt></a>.
</p>
<h3 id="part-f-cross-validation-as-resampling-techniques-adding-more-complexity">Part f): Cross-validation as resampling techniques, adding more complexity </h3>
<h3 id="part-h-cross-validation-as-resampling-techniques-adding-more-complexity">Part h): Cross-validation as resampling techniques, adding more complexity </h3>
<p>The aim here is to implement another widely popular
resampling technique, the so-called cross-validation method.
</p>
<p>Implement the \( k \)-fold cross-validation algorithm (write your own
code or use the functionality of <b>Scikit-Learn</b>) and evaluate again the MSE function resulting
<p>Implement the \( k \)-fold cross-validation algorithm (feel free to use the functionality of <b>Scikit-Learn</b> or write your own code) and evaluate again the MSE function resulting
from the test folds.
</p>
@@ -596,111 +488,7 @@ you got from your <b>bootstrap</b> code. Comment your results. Try \( 5-10 \)
folds.
</p>
<p>In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression. </p>
<h3 id="part-g-analysis-of-real-data">Part g): Analysis of real data </h3>
<p>With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
<a href="https://earthexplorer.usgs.gov/" target="_blank"><tt>https://earthexplorer.usgs.gov/</tt></a>,
</p>
<p>Or, if you prefer, we have placed selected datafiles at <a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles</tt></a></p>
<p>In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want
to fetch the digital terrain data from. In order to be able to read
the data properly, you need to specify that the format should be <b>SRTM
Arc-Second Global</b> and download the data as a <b>GeoTIF</b> file. The
files are then stored in <em>tif</em> format which can be imported into a
Python program using
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">scipy<span style="color: #666666">.</span>misc<span style="color: #666666">.</span>imread
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Here is a simple part of a Python code which reads and plots the data
from such files
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">imageio</span> <span style="color: #008000; font-weight: bold">import</span> imread
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> Axes3D
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">matplotlib</span> <span style="color: #008000; font-weight: bold">import</span> cm
<span style="color: #408080; font-style: italic"># Load the terrain</span>
terrain1 <span style="color: #666666">=</span> imread(<span style="color: #BA2121">&#39;SRTM_data_Norway_1.tif&#39;</span>)
<span style="color: #408080; font-style: italic"># Show the terrain</span>
plt<span style="color: #666666">.</span>figure()
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&#39;Terrain over Norway 1&#39;</span>)
plt<span style="color: #666666">.</span>imshow(terrain1, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&#39;gray&#39;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&#39;X&#39;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&#39;Y&#39;</span>)
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>If you should have problems in downloading the digital terrain data,
we provide two examples under the data folder of project 1. One is
from a region close to Stavanger in Norway and the other M&#248;svatn
Austfjell, again in Norway.
Feel free to produce your own terrain data.
</p>
<p>Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example <a href="https://www.kaggle.com/datasets" target="_blank">kaggle.com</a> for examples.</p>
<p>Our final part deals with the parameterization of your digital terrain
data (or your own data). We will apply all three methods for linear regression, the same type (or higher order) of polynomial
approximation and cross-validation as resampling technique to evaluate which
model fits the data best.
</p>
<p>At the end, you should present a critical evaluation of your results
and discuss the applicability of these regression methods to the type
of data presented here (either the terrain data we propose or other data sets).
</p>
<p>In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression in the analysis. </p>
<h2 id="background-literature">Background literature </h2>
<ol>