lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleaning up typos and inconsistencies in project 1

Browse Source
This commit is contained in:
mhjensen
2020-09-01 22:36:17 +02:00
parent 70dfbecd9c
commit de861c62fa
9 changed files with 231 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2020/Project1/html/._Project1-bs000.html
+45 -24
View File
@@ -45,12 +45,14 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec0'),
('Part a): Ordinary Least Square on the Franke function with '
'resampling',
('Part a): Ordinary Least Square (OLS) on the Franke function',
3,
None,
'___sec1'),
('Part b): Bias-variance trade-off', 3, None, '___sec2'),
('Part b): Bias-variance trade-off and resamplng techniques',
3,
None,
'___sec2'),
('Part c) Cross-validation as resampling techniques, adding more '
'complexity',
3,
@@ -119,8 +121,8 @@ MathJax.Hub.Config({
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;"><b>Regression analysis and resampling methods</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part a): Ordinary Least Square on the Franke function with resampling</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part b): Bias-variance trade-off</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part a): Ordinary Least Square (OLS) on the Franke function</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part b): Bias-variance trade-off and resamplng techniques</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part c) Cross-validation as resampling techniques, adding more complexity</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part d): Ridge Regression on the Franke function with resampling</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part e): Lasso Regression on the Franke function with resampling</a></li>
@@ -173,8 +175,10 @@ MathJax.Hub.Config({
<p>
The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation.
Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37.
<p>
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
<p>
We will first study how to fit polynomials to a specific
@@ -218,7 +222,7 @@ beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
<p>
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -263,7 +267,7 @@ fig<span style="color: #666666">.</span>colorbar(surf, shrink<span style="color:
plt<span style="color: #666666">.</span>show()
</pre></div>
<h3 id="___sec1" class="anchor">Part a): Ordinary Least Square on the Franke function with resampling </h3>
<h3 id="___sec1" class="anchor">Part a): Ordinary Least Square (OLS) on the Franke function </h3>
<p>
We will generate our own dataset for a function
@@ -277,7 +281,7 @@ distribution \( \cal{N}(0,1) \).
value decomposition from e.g., <b>numpy</b> ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the
confidence intervals of the parameters \( \beta \) by computing their
<a href="https://en.wikipedia.org/wiki/Confidence_interval" target="_self">confidence intervals</a> of the parameters (estimators) \( \beta \) by computing their
variances, evaluate the Mean Squared error (MSE)
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
@@ -301,9 +305,14 @@ $$
$$
<p>
Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the
function for splitting training data provided by the library <b>Scikit-Learn</b> (make sure you have installed it).
This function is called \( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of <b>Scikit-Learn</b>.
Your code has to include a scaling of the data (for example by
subtracting the mean value, see also <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_self">homework set 2</a> for examples) and
a split of the data in training and test data. For this part you can
either write your own code or use for example the function for
splitting training data provided by the library <b>Scikit-Learn</b> (make
sure you have installed it). This function is called
\( train\_test\_split \). Similarly, and see the solution to <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_self">homework set 2, exercise 2</a>, you can use the data normalization functionality of
<b>Scikit-Learn</b>.
<p>
It is normal in essentially all Machine Learning studies to split the
@@ -313,7 +322,7 @@ is no explicit recipe for how much data should be included as training
data and say test data. An accepted rule of thumb is to use
approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
<h3 id="___sec2" class="anchor">Part b): Bias-variance trade-off </h3>
<h3 id="___sec2" class="anchor">Part b): Bias-variance trade-off and resamplng techniques </h3>
<p>
Our aim here is to study the bias-variance trade-off by implementing the <b>bootstrap</b> resampling technique.
@@ -325,6 +334,16 @@ continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks and basically all Machine Learning algorithms.
<p>
Before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
indicate possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve!
<p>
With this result we move on to the bias-variance trade-off analysis.
<p>
Consider a
dataset \( \mathcal{L} \) consisting of the data
@@ -355,6 +374,8 @@ $$
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
$$
Here the expected value \( \mathbb{E} \) is the sample value.
<p>
Show that you can rewrite this as
$$
@@ -365,25 +386,25 @@ $$
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
<p>
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
<p>
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
<p>
However, before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<p>
After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the <b>bootstra</b> resampling technique.
Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<h3 id="___sec3" class="anchor">Part c) Cross-validation as resampling techniques, adding more complexity </h3>
<p>
The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method.
Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier.
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
@@ -391,7 +412,7 @@ data and test data using for example
Implement the \( k \)-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
<b>Scikit-Learn</b> if needed. You can alternatively write your own bootstrap code.
<b>Scikit-Learn</b> if needed.
<p>
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code.
doc/Projects/2020/Project1/html/Project1-bs.html
+45 -24
View File
@@ -45,12 +45,14 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec0'),
('Part a): Ordinary Least Square on the Franke function with '
'resampling',
('Part a): Ordinary Least Square (OLS) on the Franke function',
3,
None,
'___sec1'),
('Part b): Bias-variance trade-off', 3, None, '___sec2'),
('Part b): Bias-variance trade-off and resamplng techniques',
3,
None,
'___sec2'),
('Part c) Cross-validation as resampling techniques, adding more '
'complexity',
3,
@@ -119,8 +121,8 @@ MathJax.Hub.Config({
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;"><b>Regression analysis and resampling methods</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part a): Ordinary Least Square on the Franke function with resampling</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part b): Bias-variance trade-off</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part a): Ordinary Least Square (OLS) on the Franke function</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part b): Bias-variance trade-off and resamplng techniques</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part c) Cross-validation as resampling techniques, adding more complexity</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part d): Ridge Regression on the Franke function with resampling</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Part e): Lasso Regression on the Franke function with resampling</a></li>
@@ -173,8 +175,10 @@ MathJax.Hub.Config({
<p>
The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation.
Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37.
<p>
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
<p>
We will first study how to fit polynomials to a specific
@@ -218,7 +222,7 @@ beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
<p>
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -263,7 +267,7 @@ fig<span style="color: #666666">.</span>colorbar(surf, shrink<span style="color:
plt<span style="color: #666666">.</span>show()
</pre></div>
<h3 id="___sec1" class="anchor">Part a): Ordinary Least Square on the Franke function with resampling </h3>
<h3 id="___sec1" class="anchor">Part a): Ordinary Least Square (OLS) on the Franke function </h3>
<p>
We will generate our own dataset for a function
@@ -277,7 +281,7 @@ distribution \( \cal{N}(0,1) \).
value decomposition from e.g., <b>numpy</b> ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the
confidence intervals of the parameters \( \beta \) by computing their
<a href="https://en.wikipedia.org/wiki/Confidence_interval" target="_self">confidence intervals</a> of the parameters (estimators) \( \beta \) by computing their
variances, evaluate the Mean Squared error (MSE)
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
@@ -301,9 +305,14 @@ $$
$$
<p>
Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the
function for splitting training data provided by the library <b>Scikit-Learn</b> (make sure you have installed it).
This function is called \( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of <b>Scikit-Learn</b>.
Your code has to include a scaling of the data (for example by
subtracting the mean value, see also <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_self">homework set 2</a> for examples) and
a split of the data in training and test data. For this part you can
either write your own code or use for example the function for
splitting training data provided by the library <b>Scikit-Learn</b> (make
sure you have installed it). This function is called
\( train\_test\_split \). Similarly, and see the solution to <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_self">homework set 2, exercise 2</a>, you can use the data normalization functionality of
<b>Scikit-Learn</b>.
<p>
It is normal in essentially all Machine Learning studies to split the
@@ -313,7 +322,7 @@ is no explicit recipe for how much data should be included as training
data and say test data. An accepted rule of thumb is to use
approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
<h3 id="___sec2" class="anchor">Part b): Bias-variance trade-off </h3>
<h3 id="___sec2" class="anchor">Part b): Bias-variance trade-off and resamplng techniques </h3>
<p>
Our aim here is to study the bias-variance trade-off by implementing the <b>bootstrap</b> resampling technique.
@@ -325,6 +334,16 @@ continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks and basically all Machine Learning algorithms.
<p>
Before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
indicate possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve!
<p>
With this result we move on to the bias-variance trade-off analysis.
<p>
Consider a
dataset \( \mathcal{L} \) consisting of the data
@@ -355,6 +374,8 @@ $$
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
$$
Here the expected value \( \mathbb{E} \) is the sample value.
<p>
Show that you can rewrite this as
$$
@@ -365,25 +386,25 @@ $$
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
<p>
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
<p>
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
<p>
However, before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<p>
After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the <b>bootstra</b> resampling technique.
Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<h3 id="___sec3" class="anchor">Part c) Cross-validation as resampling techniques, adding more complexity </h3>
<p>
The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method.
Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier.
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
@@ -391,7 +412,7 @@ data and test data using for example
Implement the \( k \)-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
<b>Scikit-Learn</b> if needed. You can alternatively write your own bootstrap code.
<b>Scikit-Learn</b> if needed.
<p>
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code.
doc/Projects/2020/Project1/html/Project1.html
+43 -22
View File
@@ -44,12 +44,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec0'),
('Part a): Ordinary Least Square on the Franke function with '
'resampling',
('Part a): Ordinary Least Square (OLS) on the Franke function',
3,
None,
'___sec1'),
('Part b): Bias-variance trade-off', 3, None, '___sec2'),
('Part b): Bias-variance trade-off and resamplng techniques',
3,
None,
'___sec2'),
('Part c) Cross-validation as resampling techniques, adding more '
'complexity',
3,
@@ -128,8 +130,10 @@ MathJax.Hub.Config({
<p>
The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation.
Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37.
<p>
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
<p>
We will first study how to fit polynomials to a specific
@@ -173,7 +177,7 @@ beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
<p>
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -218,7 +222,7 @@ fig<span style="color: #666666">.</span>colorbar(surf, shrink<span style="color:
plt<span style="color: #666666">.</span>show()
</pre></div>
<h3 id="___sec1">Part a): Ordinary Least Square on the Franke function with resampling </h3>
<h3 id="___sec1">Part a): Ordinary Least Square (OLS) on the Franke function </h3>
<p>
We will generate our own dataset for a function
@@ -232,7 +236,7 @@ distribution \( \cal{N}(0,1) \).
value decomposition from e.g., <b>numpy</b> ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the
confidence intervals of the parameters \( \beta \) by computing their
<a href="https://en.wikipedia.org/wiki/Confidence_interval" target="_blank">confidence intervals</a> of the parameters (estimators) \( \beta \) by computing their
variances, evaluate the Mean Squared error (MSE)
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
@@ -256,9 +260,14 @@ $$
$$
<p>
Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the
function for splitting training data provided by the library <b>Scikit-Learn</b> (make sure you have installed it).
This function is called \( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of <b>Scikit-Learn</b>.
Your code has to include a scaling of the data (for example by
subtracting the mean value, see also <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_blank">homework set 2</a> for examples) and
a split of the data in training and test data. For this part you can
either write your own code or use for example the function for
splitting training data provided by the library <b>Scikit-Learn</b> (make
sure you have installed it). This function is called
\( train\_test\_split \). Similarly, and see the solution to <a href="https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" target="_blank">homework set 2, exercise 2</a>, you can use the data normalization functionality of
<b>Scikit-Learn</b>.
<p>
It is normal in essentially all Machine Learning studies to split the
@@ -268,7 +277,7 @@ is no explicit recipe for how much data should be included as training
data and say test data. An accepted rule of thumb is to use
approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
<h3 id="___sec2">Part b): Bias-variance trade-off </h3>
<h3 id="___sec2">Part b): Bias-variance trade-off and resamplng techniques </h3>
<p>
Our aim here is to study the bias-variance trade-off by implementing the <b>bootstrap</b> resampling technique.
@@ -280,6 +289,16 @@ continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks and basically all Machine Learning algorithms.
<p>
Before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
indicate possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve!
<p>
With this result we move on to the bias-variance trade-off analysis.
<p>
Consider a
dataset \( \mathcal{L} \) consisting of the data
@@ -310,6 +329,8 @@ $$
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
$$
Here the expected value \( \mathbb{E} \) is the sample value.
<p>
Show that you can rewrite this as
$$
@@ -320,25 +341,25 @@ $$
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
<p>
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
<p>
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
<p>
However, before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<p>
After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the <b>bootstra</b> resampling technique.
Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
<h3 id="___sec3">Part c) Cross-validation as resampling techniques, adding more complexity </h3>
<p>
The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method.
Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier.
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
@@ -346,7 +367,7 @@ data and test data using for example
Implement the \( k \)-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
<b>Scikit-Learn</b> if needed. You can alternatively write your own bootstrap code.
<b>Scikit-Learn</b> if needed.
<p>
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code.
doc/Projects/2020/Project1/ipynb/ipynb-Project1-src.tar.gz
BIN
View File
Binary file not shown.
doc/Projects/2020/Project1/pdf/Project1.p.tex
+32 -17
View File
@@ -166,8 +166,9 @@ Sep 1, 2020
The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation.
Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
We will first study how to fit polynomials to a specific
two-dimensional function called \href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's
@@ -206,7 +207,7 @@ beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
\bpycod
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
@@ -251,7 +252,7 @@ plt.show()
\epycod
\paragraph{Part a): Ordinary Least Square on the Franke function with resampling.}
\paragraph{Part a): Ordinary Least Square (OLS) on the Franke function.}
We will generate our own dataset for a function
$\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function
$f(x,y)$ is the Franke function. You should explore also the addition
@@ -262,7 +263,7 @@ distribution $\cal{N}(0,1)$.
value decomposition from e.g., \textbf{numpy} ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in $x$ and $y$ up to fifth order. Find the
confidence intervals of the parameters $\beta$ by computing their
\href{{https://en.wikipedia.org/wiki/Confidence_interval}}{confidence intervals} of the parameters (estimators) $\beta$ by computing their
variances, evaluate the Mean Squared error (MSE)
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
@@ -283,9 +284,14 @@ where we have defined the mean value of $\hat{y}$ as
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
\]
Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the
function for splitting training data provided by the library \textbf{Scikit-Learn} (make sure you have installed it).
This function is called $train\_test\_split$. Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of \textbf{Scikit-Learn}.
Your code has to include a scaling of the data (for example by
subtracting the mean value, see also \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2} for examples) and
a split of the data in training and test data. For this part you can
either write your own code or use for example the function for
splitting training data provided by the library \textbf{Scikit-Learn} (make
sure you have installed it). This function is called
$train\_test\_split$. Similarly, and see the solution to \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2, exercise 2}, you can use the data normalization functionality of
\textbf{Scikit-Learn}.
It is normal in essentially all Machine Learning studies to split the
data in a training set and a test set (eventually also an additional
@@ -299,7 +305,7 @@ approximately $2/3$ to $4/5$ of the data as training data.
\paragraph{Part b): Bias-variance trade-off.}
\paragraph{Part b): Bias-variance trade-off and resamplng techniques.}
Our aim here is to study the bias-variance trade-off by implementing the \textbf{bootstrap} resampling technique.
With a code which does OLS and includes resampling techniques,
@@ -308,6 +314,14 @@ continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks and basically all Machine Learning algorithms.
Before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
indicate possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve!
With this result we move on to the bias-variance trade-off analysis.
Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
@@ -332,6 +346,7 @@ squared error via the so-called cost function
\[
C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
\]
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this as
\[
@@ -341,21 +356,21 @@ Show that you can rewrite this as
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method.
However, before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the \textbf{bootstra} resampling technique.
\paragraph{Part c) Cross-validation as resampling techniques, adding more complexity.}
The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method.
Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier.
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
@@ -364,7 +379,7 @@ data and test data using for example
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
\textbf{Scikit-Learn} if needed. You can alternatively write your own bootstrap code.
\textbf{Scikit-Learn} if needed.
Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code.
You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}.
doc/Projects/2020/Project1/pdf/Project1.pdf
BIN
View File
Binary file not shown.
doc/Projects/2020/Project1/pdf/Project1.tex
+33 -18
View File
@@ -136,8 +136,9 @@ Sep 1, 2020
The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation.
Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
We will first study how to fit polynomials to a specific
two-dimensional function called \href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's
@@ -176,7 +177,7 @@ beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
\begin{verbatim}
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
@@ -221,7 +222,7 @@ plt.show()
\end{verbatim}
\paragraph{Part a): Ordinary Least Square on the Franke function with resampling.}
\paragraph{Part a): Ordinary Least Square (OLS) on the Franke function.}
We will generate our own dataset for a function
$\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function
$f(x,y)$ is the Franke function. You should explore also the addition
@@ -232,7 +233,7 @@ distribution $\cal{N}(0,1)$.
value decomposition from e.g., \textbf{numpy} ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in $x$ and $y$ up to fifth order. Find the
confidence intervals of the parameters $\beta$ by computing their
\href{{https://en.wikipedia.org/wiki/Confidence_interval}}{confidence intervals} of the parameters (estimators) $\beta$ by computing their
variances, evaluate the Mean Squared error (MSE)
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
@@ -253,9 +254,14 @@ where we have defined the mean value of $\hat{y}$ as
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
\]
Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the
function for splitting training data provided by the library \textbf{Scikit-Learn} (make sure you have installed it).
This function is called $train\_test\_split$. Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of \textbf{Scikit-Learn}.
Your code has to include a scaling of the data (for example by
subtracting the mean value, see also \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2} for examples) and
a split of the data in training and test data. For this part you can
either write your own code or use for example the function for
splitting training data provided by the library \textbf{Scikit-Learn} (make
sure you have installed it). This function is called
$train\_test\_split$. Similarly, and see the solution to \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2, exercise 2}, you can use the data normalization functionality of
\textbf{Scikit-Learn}.
It is normal in essentially all Machine Learning studies to split the
data in a training set and a test set (eventually also an additional
@@ -269,7 +275,7 @@ approximately $2/3$ to $4/5$ of the data as training data.
\paragraph{Part b): Bias-variance trade-off.}
\paragraph{Part b): Bias-variance trade-off and resamplng techniques.}
Our aim here is to study the bias-variance trade-off by implementing the \textbf{bootstrap} resampling technique.
With a code which does OLS and includes resampling techniques,
@@ -278,6 +284,14 @@ continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks and basically all Machine Learning algorithms.
Before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
indicate possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve!
With this result we move on to the bias-variance trade-off analysis.
Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
@@ -302,6 +316,7 @@ squared error via the so-called cost function
\[
C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
\]
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this as
\[
@@ -311,21 +326,21 @@ Show that you can rewrite this as
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method.
However, before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the \textbf{bootstra} resampling technique.
\paragraph{Part c) Cross-validation as resampling techniques, adding more complexity.}
The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method.
Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier.
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
@@ -334,7 +349,7 @@ data and test data using for example
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
\textbf{Scikit-Learn} if needed. You can alternatively write your own bootstrap code.
\textbf{Scikit-Learn} if needed.
Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code.
You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}.
@@ -408,7 +423,7 @@ Austfjell, again in Norway.
Feel free to produce your own terrain data.
Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your research area or simply a data set you found interesting. See for example \href{{https://www.kaggle.com/datasets}}{kaggle.com} for examples.
Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example \href{{https://www.kaggle.com/datasets}}{kaggle.com} for examples.
\paragraph{Part g) OLS, Ridge and Lasso regression with resampling.}
Our final part deals with the parameterization of your digital terrain
doc/src/Projects/2020/Project1/Project1.do.txt
+32 -17
View File
@@ -8,8 +8,9 @@ DATE: today
The main aim of this project is to study in more detail various
regression methods, including the Ordinary Least Squares (OLS) method,
Ridge regression and finally Lasso regression.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation.
Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37.
The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
We will first study how to fit polynomials to a specific
two-dimensional function called "Franke's
@@ -50,7 +51,7 @@ beyond the second-order polynomials metioned above and explore
which polynomial fits the data best.
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
!bc pycod
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
@@ -95,7 +96,7 @@ plt.show()
!ec
=== Part a): Ordinary Least Square on the Franke function with resampling ===
=== Part a): Ordinary Least Square (OLS) on the Franke function ===
We will generate our own dataset for a function
$\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function
@@ -107,7 +108,7 @@ distribution $\cal{N}(0,1)$.
value decomposition from e.g., _numpy_ ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in $x$ and $y$ up to fifth order. Find the
confidence intervals of the parameters $\beta$ by computing their
"confidence intervals":"https://en.wikipedia.org/wiki/Confidence_interval" of the parameters (estimators) $\beta$ by computing their
variances, evaluate the Mean Squared error (MSE)
!bt
@@ -134,9 +135,14 @@ where we have defined the mean value of $\hat{y}$ as
\]
!et
Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the
function for splitting training data provided by the library _Scikit-Learn_ (make sure you have installed it).
This function is called $train\_test\_split$. Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of _Scikit-Learn_.
Your code has to include a scaling of the data (for example by
subtracting the mean value, see also "homework set 2":"https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" for examples) and
a split of the data in training and test data. For this part you can
either write your own code or use for example the function for
splitting training data provided by the library _Scikit-Learn_ (make
sure you have installed it). This function is called
$train\_test\_split$. Similarly, and see the solution to "homework set 2, exercise 2":"https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html", you can use the data normalization functionality of
_Scikit-Learn_.
It is normal in essentially all Machine Learning studies to split the
data in a training set and a test set (eventually also an additional
@@ -150,7 +156,7 @@ approximately $2/3$ to $4/5$ of the data as training data.
=== Part b): Bias-variance trade-off ===
=== Part b): Bias-variance trade-off and resamplng techniques ===
Our aim here is to study the bias-variance trade-off by implementing the _bootstrap_ resampling technique.
@@ -160,6 +166,14 @@ continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks and basically all Machine Learning algorithms.
Before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
indicate possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve!
With this result we move on to the bias-variance trade-off analysis.
Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
@@ -188,6 +202,7 @@ squared error via the so-called cost function
C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
\]
!et
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this as
!bt
@@ -199,23 +214,23 @@ Show that you can rewrite this as
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the _bootstrap_ resampling method.
However, before you perform an analysis of the bias-variance trade-off on your test data, make
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an
equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the _bootstra_ resampling technique.
=== Part c) Cross-validation as resampling techniques, adding more complexity ===
The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method.
Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier.
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
@@ -224,7 +239,7 @@ data and test data using for example
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
_Scikit-Learn_ if needed. You can alternatively write your own bootstrap code.
_Scikit-Learn_ if needed.
Compare the MSE you get from your cross-validation code with the one you got from your _bootstrap_ code.
You can also compare your own cross-validation code with the one provided by _Scikit-Learn_.
doc/src/Projects/2020/Project1/make.sh
+1 -1
View File
@@ -38,7 +38,7 @@ system doconce split_html $html.html --method=split --pagination --nav_button=bo
# Ordinary plain LaTeX document
system doconce format pdflatex $name --print_latex_style=trac --latex_admon=paragraph $opt
system doconce ptex2tex $name envir=print
system doconce ptex2tex $name envir=verbatim
# Add special packages
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
doconce replace 'section{' 'section*{' $name.tex