lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
00a2bd1a7efde5fbfd9b9d6d6c365dcd82fe8baf
FYS-STK4155/doc/src/week36/._exercisesweek36-bs000.html
T
Morten Hjorth-Jensen 195eced7a4 small update
2023-09-05 05:57:31 +02:00

286 lines
12 KiB
HTML
Raw Blame History

<!--
HTML file automatically generated from DocOnce source
(https://github.com/doconce/doconce/)
doconce format html exercisesweek36.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=exercisesweek36-bs --no_mako
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Exercises week 36">
<title>Exercises week 36</title>
<!-- Bootstrap style: bootstrap -->
<!-- doconce format html exercisesweek36.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=exercisesweek36-bs --no_mako -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Overarching aims of the exercises this week',
2,
None,
'overarching-aims-of-the-exercises-this-week'),
('Exercise 1: Analytical exercises',
2,
None,
'exercise-1-analytical-exercises'),
('a) Expression for Ridge regression',
3,
None,
'a-expression-for-ridge-regression'),
('b) The singular value decomposition',
3,
None,
'b-the-singular-value-decomposition'),
('Exercise 2: Adding Ridge Regression',
2,
None,
'exercise-2-adding-ridge-regression')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="exercisesweek36-bs.html">Exercises week 36</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#overarching-aims-of-the-exercises-this-week" style="font-size: 80%;"><b>Overarching aims of the exercises this week</b></a></li>
<!-- navigation toc: --> <li><a href="#exercise-1-analytical-exercises" style="font-size: 80%;"><b>Exercise 1: Analytical exercises</b></a></li>
<!-- navigation toc: --> <li><a href="#a-expression-for-ridge-regression" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;a) Expression for Ridge regression</a></li>
<!-- navigation toc: --> <li><a href="#b-the-singular-value-decomposition" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;b) The singular value decomposition</a></li>
<!-- navigation toc: --> <li><a href="#exercise-2-adding-ridge-regression" style="font-size: 80%;"><b>Exercise 2: Adding Ridge Regression</b></a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0000"></a>
<!-- ------------------- main content ---------------------- -->
<div class="jumbotron">
<center>
<h1>Exercises week 36</h1>
</center> <!-- document title -->
<!-- author(s): September 4-8, 2023 -->
<center>
<b>September 4-8, 2023</b>
</center>
<!-- institution(s) -->
<br>
<center>
<h4>Deadline is Sunday September 10 at midnight</h4>
</center> <!-- date -->
<br>
</div> <!-- end jumbotron -->
<h2 id="overarching-aims-of-the-exercises-this-week" class="anchor">Overarching aims of the exercises this week </h2>
<p>This set of exercises form an important part of the first project. The
analytical exercises deal with the material covered last week on the
mathematical interpretations of ordinary least squares and of Ridge
regression. The numerical exercises can be seen as a continuation of
exercise 3 from week 35, with the inclusion of Ridge regression. This
material enters also the discussions of the first project.
</p>
<!-- --- begin exercise --- -->
<h2 id="exercise-1-analytical-exercises" class="anchor">Exercise 1: Analytical exercises </h2>
<p>The aim here is to derive the expression for the optimal parameters
using Ridge regression. Furthermore, using the singular value
decomposition, we will analyze the difference between the ordinary
least squares approach and Ridge regression.
</p>
<p>The expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, was given by the
optimization problem
</p>
$$
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
$$
<p>which we can also write as</p>
$$
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
$$
<p>where we have used the definition of a norm-2 vector, that is</p>
$$
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
$$
<p>By minimizing the above equation with respect to the parameters
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
parameters \( \boldsymbol{\beta} \).
</p>
<p>We can add a regularization parameter \( \lambda \) by
defining a new cost function to be optimized, that is
</p>
$$
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
$$
<p>which leads to the Ridge regression minimization problem. One can require as part of the optimization problem
that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
a finite number larger than zero. We will not implement that here.
</p>
<h3 id="a-expression-for-ridge-regression" class="anchor">a) Expression for Ridge regression </h3>
<p>Show that the optimal parameters</p>
$$
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<p>with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that</p>
$$
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
$$
<p>with \( t \) a finite positive number. In the optimization, we will not require that the latter is satisfied.</p>
<p>The ordinary least squares result is</p>
$$
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<h3 id="b-the-singular-value-decomposition" class="anchor">b) The singular value decomposition </h3>
<p>Here we will use the singular value decomposition of an \( n\times p \) matrix \( \boldsymbol{X} \) (our design matrix)</p>
$$
\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
$$
<p>to study properties of Ridge regression and ordinary least squares regression.
Here \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal matrices of dimensions
\( n\times n \) and \( p\times p \), respectively, and \( \boldsymbol{\Sigma} \) is an
\( n\times p \) matrix which contains the singular values only. This material was discussed during the lectures of week 35.
</p>
<p>Show that you can write the
OLS solutions in terms of the eigenvectors (the columns) of the orthogonal matrix \( \boldsymbol{U} \) as
</p>
$$
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} = \sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}.
$$
<p>For Ridge regression, show that the corresponding equation is</p>
$$
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
<p>with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). </p>
<p>Give an interpretation of the results. <a href="https://link.springer.com/book/10.1007/978-0-387-84858-7" target="_self">Section 3.4 of Hastie et al's textbook gives a good discussion of the above results</a>.</p>
<!-- --- end exercise --- -->
<!-- --- begin exercise --- -->
<h2 id="exercise-2-adding-ridge-regression" class="anchor">Exercise 2: Adding Ridge Regression </h2>
<p>This exercise is a continuation of exercise 3 from week 35, see <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek35.html" target="_self"><tt>https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek35.html</tt></a>. We will use the same function to
generate our data set, still staying with a simple function \( y(x) \)
which we want to fit using linear regression, but now extending the
analysis to include the Ridge regression method.
</p>
<p>In this exercise you need to include the same elements from last week, that is</p>
<ol>
<li> scale your data by subtracting the mean value from each column in the design matrix.</li>
<li> perform a split of the data in a training set and a test set.</li>
</ol>
<p>The addition to the analysis this time is the introduction of the hyperparameter \( \lambda \) when introducing Ridge regression.</p>
<p>Extend the code from exercise 3 from <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek35.html" target="_self">week 35</a> to include Ridge regression with the hyperparameter \( \lambda \). The optimal parameters \( \hat{\beta} \) for Ridge regression can be obtained by matrix inversion in a similar way as done for ordinary least squares. You need to add to your code the following equations</p>
$$
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>The ordinary least squares result you encoded last week is given by</p>
$$
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<p>Use these results to compute the mean squared error for ordinary least
squares and Ridge regression first for a polynomial of degree five
with \( n=100 \) data points and five selected values of
\( \lambda=[0.0001,0.001, 0.01,0.1,1.0] \). Compute thereafter the mean
squared error for the same values of \( \lambda \) for polynomials of degree ten
and \( 15 \). Discuss your results for the training MSE and test MSE with
Ridge regression and ordinary least squares.
</p>
<!-- --- end exercise --- -->
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li class="active"><a href="._exercisesweek36-bs000.html">1</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink