mhjensen
161 lines
7.3 KiB
HTML
161 lines
7.3 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="description" content="Homework 2">
|
|
|
|
<title>Homework 2</title>
|
|
|
|
|
|
<style type="text/css">
|
|
/* bloodish style */
|
|
|
|
body {
|
|
font-family: Helvetica, Verdana, Arial, Sans-serif;
|
|
color: #404040;
|
|
background: #ffffff;
|
|
}
|
|
h1 { font-size: 1.8em; color: #8A0808; }
|
|
h2 { font-size: 1.6em; color: #8A0808; }
|
|
h3 { font-size: 1.4em; color: #8A0808; }
|
|
h4 { color: #8A0808; }
|
|
a { color: #8A0808; text-decoration:none; }
|
|
tt { font-family: "Courier New", Courier; }
|
|
/* pre style removed because it will interfer with pygments */
|
|
p { text-indent: 0px; }
|
|
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
p.caption { width: 80%; font-style: normal; text-align: left; }
|
|
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
|
|
div { text-align: justify; text-justify: inter-word; }
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Exercise 4', 2, None, '___sec0'),
|
|
('Exercise 5', 2, None, '___sec1')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: {
|
|
equationNumbers: { autoNumber: "AMS" },
|
|
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
|
}
|
|
});
|
|
</script>
|
|
<script type="text/javascript" async
|
|
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
|
</script>
|
|
|
|
|
|
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<center><h1>Homework 2</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): <a href="http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" target="_blank">Data Analysis and Machine Learning FYS-STK3155/FYS4155</a> -->
|
|
|
|
<center>
|
|
<b><a href="http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" target="_blank">Data Analysis and Machine Learning FYS-STK3155/FYS4155</a></b>
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution -->
|
|
|
|
<center><b>Department of Physics, University of Oslo, Norway</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Sep 5, 2018</h4></center> <!-- date -->
|
|
<br>
|
|
|
|
<h2 id="___sec0">Exercise 4 </h2>
|
|
|
|
<p>
|
|
This exercise is a continuation of exercise 2 from homework 1. We will
|
|
use the same function to generate our data set, still staying with a
|
|
simple function \( y(x) \) which we want to fit using linear regression,
|
|
but now extending the analysis to include the Ridge and the Lasso
|
|
regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the <a href="https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html" target="_blank">regression slides</a>).
|
|
|
|
<p>
|
|
We will thus again generate our own dataset for a function \( y(x) \) where
|
|
\( x \in [0,1] \) and defined by random numbers computed with the uniform
|
|
distribution. The function \( y \) is a quadratic polynomial in \( x \) with
|
|
added stochastic noise according to the normal distribution \( \cal{N}(0,1) \).
|
|
|
|
<p>
|
|
The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
|
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
|
y <span style="color: #666666">=</span> <span style="color: #666666">5*</span>x<span style="color: #666666">*</span>x<span style="color: #666666">+0.1*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
|
</pre></div>
|
|
<ol>
|
|
<li> Write your own code for the Ridge method (see chapter 3.4 of Hastie <em>et al.</em>, equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 2. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).</li>
|
|
<li> Repeat the above but using the functionality of <b>scikit-learn</b>. Compare your code with the results from <b>scikit-learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
|
|
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of <b>scikit-learn</b> and compute their variances. Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie <em>et al.</em> and their figure 3.11.</li>
|
|
<li> Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie <em>et al.</em>. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of <b>scikit-learn</b>.</li>
|
|
<li> Finally, using <b>scikit-learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
|
|
</ol>
|
|
|
|
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
|
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
|
$$
|
|
|
|
and the \( R^2 \) score function.
|
|
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
|
|
$$
|
|
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
|
$$
|
|
|
|
where we have defined the mean value of \( \hat{y} \) as
|
|
$$
|
|
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
|
$$
|
|
|
|
Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods.
|
|
|
|
<h2 id="___sec1">Exercise 5 </h2>
|
|
|
|
<p>
|
|
Using the singular value decomposition, show that the variance of the direction vector
|
|
\( \hat{z}_i=\hat{X}\hat{v}_i=\hat{u}_1d_1 \) is equal to (equation (3.49) of Hastie <em>et al.</em>)
|
|
$$
|
|
\mathrm{Var}(\hat{z}_i)=\frac{d_i^2}{N},
|
|
$$
|
|
|
|
where \( d_i \) are the singular values of the matrix \( \hat{X} \). In Hastie <em>et al</em>, the matrix elements of \( X \) are centered. The consequence is that the mean values of for example \( \hat{u}_i \) are zero.
|
|
|
|
<p>
|
|
Give an interpretation of these results, in particular in connection with the variance of the coefficients you obtained in the previous exercise.
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|