lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleaning up typos and inconsistencies in project 1

Browse Source
This commit is contained in:
mhjensen
2020-09-01 22:36:17 +02:00
parent 70dfbecd9c
commit de861c62fa
9 changed files with 231 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2020/Project1/html/Project1.html
+43 -22
View File
@@ -44,12 +44,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec0'),
('Part a): Ordinary Least Square on the Franke function with '
'resampling',
('Part a): Ordinary Least Square (OLS) on the Franke function',
3,
None,
'___sec1'),
('Part b): Bias-variance trade-off', 3, None, '___sec2'),
('Part b): Bias-variance trade-off and resamplng techniques',
3,
None,
'___sec2'),
('Part c) Cross-validation as resampling techniques, adding more '
'complexity',
3,
@@ -128,8 +130,10 @@ MathJax.Hub.Config({
<p>
The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation.
Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37.
<p>
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
<p>
We will first study how to fit polynomials to a specific
@@ -173,7 +177,7 @@ beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
<p>
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -218,7 +222,7 @@ fig<span style="color: #666666">.</span>colorbar(surf, shrink<span style="color:
plt<span style="color: #666666">.</span>show()
</pre></div>
<h3 id="___sec1">Part a): Ordinary Least Square on the Franke function with resampling </h3>
<h3 id="___sec1">Part a): Ordinary Least Square (OLS) on the Franke function </h3>
<p>
We will generate our own dataset for a function
@@ -232,7 +236,7 @@ distribution \( \cal{N}(0,1) \).
value decomposition from e.g., <b>numpy</b> ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the
confidence intervals of the parameters \( \beta \) by computing their
<a href="https://en.wikipedia.org/wiki/Confidence_interval" target="_blank">confidence intervals</a> of the parameters (estimators) \( \beta \) by computing their
variances, evaluate the Mean Squared error (MSE)
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
@@ -256,9 +260,14 @@ $$
$$
<p>
Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the
function for splitting training data provided by the library <b>Scikit-Learn</b> (make sure you have installed it).
This function is called \( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of <b>Scikit-Learn</b>.
Your code has to include a scaling of the data (for example by
subtracting the mean value, see also <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_blank">homework set 2</a> for examples) and
a split of the data in training and test data. For this part you can
either write your own code or use for example the function for
splitting training data provided by the library <b>Scikit-Learn</b> (make
sure you have installed it). This function is called
\( train\_test\_split \). Similarly, and see the solution to <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_blank">homework set 2, exercise 2</a>, you can use the data normalization functionality of
<b>Scikit-Learn</b>.
<p>
It is normal in essentially all Machine Learning studies to split the
@@ -268,7 +277,7 @@ is no explicit recipe for how much data should be included as training
data and say test data. An accepted rule of thumb is to use
approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
<h3 id="___sec2">Part b): Bias-variance trade-off </h3>
<h3 id="___sec2">Part b): Bias-variance trade-off and resamplng techniques </h3>
<p>
Our aim here is to study the bias-variance trade-off by implementing the <b>bootstrap</b> resampling technique.
@@ -280,6 +289,16 @@ continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks and basically all Machine Learning algorithms.
<p>
Before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
indicate possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve!
<p>
With this result we move on to the bias-variance trade-off analysis.
<p>
Consider a
dataset \( \mathcal{L} \) consisting of the data
@@ -310,6 +329,8 @@ $$
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
$$
Here the expected value \( \mathbb{E} \) is the sample value.
<p>
Show that you can rewrite this as
$$
@@ -320,25 +341,25 @@ $$
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
<p>
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
<p>
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
<p>
However, before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<p>
After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the <b>bootstra</b> resampling technique.
Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<h3 id="___sec3">Part c) Cross-validation as resampling techniques, adding more complexity </h3>
<p>
The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method.
Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier.
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
@@ -346,7 +367,7 @@ data and test data using for example
Implement the \( k \)-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
<b>Scikit-Learn</b> if needed. You can alternatively write your own bootstrap code.
<b>Scikit-Learn</b> if needed.
<p>
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code.