diff --git a/doc/Projects/2023/Project1/html/._Project1-bs000.html b/doc/Projects/2023/Project1/html/._Project1-bs000.html
index e4b893262..3a8cec6ec 100644
--- a/doc/Projects/2023/Project1/html/._Project1-bs000.html
+++ b/doc/Projects/2023/Project1/html/._Project1-bs000.html
@@ -48,24 +48,27 @@ doconce format html Project1.do.txt --html_style=bootstrap --pygments_html_style
3,
None,
'part-a-ordinary-least-square-ols-on-the-franke-function'),
- ('Part b): Adding Ridge and Lasso Regression on the Franke '
- 'function',
+ ('Part b): Adding Ridge regression for the Franke function',
3,
None,
- 'part-b-adding-ridge-and-lasso-regression-on-the-franke-function'),
- ('Part a): Paper and pencil part',
+ 'part-b-adding-ridge-regression-for-the-franke-function'),
+ ('Part c): Adding Lasso for the Franke function',
3,
None,
- 'part-a-paper-and-pencil-part'),
- ('Part c): Bias-variance trade-off and resampling techniques',
+ 'part-c-adding-lasso-for-the-franke-function'),
+ ('Part d): Paper and pencil part',
3,
None,
- 'part-c-bias-variance-trade-off-and-resampling-techniques'),
- ('Part d): Cross-validation as resampling techniques, adding '
+ 'part-d-paper-and-pencil-part'),
+ ('Part e): Bias-variance trade-off and resampling techniques',
+ 3,
+ None,
+ 'part-e-bias-variance-trade-off-and-resampling-techniques'),
+ ('Part f): Cross-validation as resampling techniques, adding '
'more complexity',
3,
None,
- 'part-d-cross-validation-as-resampling-techniques-adding-more-complexity'),
+ 'part-f-cross-validation-as-resampling-techniques-adding-more-complexity'),
('Part g): Analysis of real data',
3,
None,
@@ -120,10 +123,11 @@ MathJax.Hub.Config({
Regression analysis and resampling methods
Description of two-dimensional function
Part a) : Ordinary Least Square (OLS) on the Franke function
- Part b): Adding Ridge and Lasso Regression on the Franke function
- Part a): Paper and pencil part
- Part c): Bias-variance trade-off and resampling techniques
- Part d): Cross-validation as resampling techniques, adding more complexity
+ Part b): Adding Ridge regression for the Franke function
+ Part c): Adding Lasso for the Franke function
+ Part d): Paper and pencil part
+ Part e): Bias-variance trade-off and resampling techniques
+ Part f): Cross-validation as resampling techniques, adding more complexity
Part g): Analysis of real data
Background literature
Introduction to numerical projects
@@ -173,14 +177,14 @@ for example
we detail how to write a report. Furthermore, at
https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/
you can find examples of previous reports. How to write reports will
-also be discussed during lectures and at the various lab sessions.
+also be discussed during the various lab sessions.
A small recommendation when developing the codes here. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
-of week 35. A simple test, as discussed during the lectures the first
+of weeks 35 and 36. A simple test, as discussed during the lectures the first
two weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
@@ -212,12 +216,17 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}
\end{align*}
$$
-
The function will be defined for \( x,y\in [0,1] \). Our first step will
+
The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.
+
+Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y,
x^2, y^2, xy, \dots] \). We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
-technique. As discussed in the lectures for weeks 35 and 36,, we can
+technique.
+
+
+We can
use a uniform distribution to set up the arrays of values for \( x \) and
\( y \), or as in the example below just a set of fixed values for \( x \) and
\( y \) with a given step size. We will fit a function (for example a
@@ -352,26 +361,29 @@ approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
-Part b): Adding Ridge and Lasso Regression on the Franke function
+Part b): Adding Ridge regression for the Franke function
Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
-exercise.
+exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and
-analyze your results with those obtained in parts b-d). Study the
+analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on \( \lambda \).
+Part c): Adding Lasso for the Franke function
This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
-you can also use the functionalities of Scikit-Learn (recommended).
+you can also use the functionalities of Scikit-Learn (recommended). Keep in mind that the library Scikit-Learn excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
-model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the bootstrap resampling technique and an analysis of the mean squared error using cross-validation.
+model fits the data best.
-Part a): Paper and pencil part
+
+v
+Part d): Paper and pencil part
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
@@ -421,9 +433,11 @@ $$
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). .
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
-Part c): Bias-variance trade-off and resampling techniques
+Part e): Bias-variance trade-off and resampling techniques
-Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+We will only use the simpler ordinary least squares here.
+
With a code which does OLS and includes resampling techniques,
we will now discuss the bias-variance trade-off in the context of
@@ -446,7 +460,7 @@ dataset \( \mathcal{L} \) consisting of the data
\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
$$
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
@@ -503,12 +517,10 @@ of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the bootstrap resampling method.
You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff.
-Part d): Cross-validation as resampling techniques, adding more complexity
+Part f): Cross-validation as resampling techniques, adding more complexity
The aim here is to write your own code for another widely popular
-resampling technique, the so-called cross-validation method. Again,
-before you start with cross-validation approach, you should scale your
-data if you think this is needed.
+resampling technique, the so-called cross-validation method.
Implement the \( k \)-fold cross-validation algorithm (write your own
@@ -522,17 +534,19 @@ you got from your bootstrap code. Comment your results. Try \( 5-10 \)
folds. You can also compare your own cross-validation code with the
one provided by Scikit-Learn.
+
+In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
Part g): Analysis of real data
With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
-essentially repeat in this exercise what was done in exercises 1-5. However, we
+essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
https://earthexplorer.usgs.gov/,
-Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles
+Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles
In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want
diff --git a/doc/Projects/2023/Project1/html/Project1-bs.html b/doc/Projects/2023/Project1/html/Project1-bs.html
index e4b893262..3a8cec6ec 100644
--- a/doc/Projects/2023/Project1/html/Project1-bs.html
+++ b/doc/Projects/2023/Project1/html/Project1-bs.html
@@ -48,24 +48,27 @@ doconce format html Project1.do.txt --html_style=bootstrap --pygments_html_style
3,
None,
'part-a-ordinary-least-square-ols-on-the-franke-function'),
- ('Part b): Adding Ridge and Lasso Regression on the Franke '
- 'function',
+ ('Part b): Adding Ridge regression for the Franke function',
3,
None,
- 'part-b-adding-ridge-and-lasso-regression-on-the-franke-function'),
- ('Part a): Paper and pencil part',
+ 'part-b-adding-ridge-regression-for-the-franke-function'),
+ ('Part c): Adding Lasso for the Franke function',
3,
None,
- 'part-a-paper-and-pencil-part'),
- ('Part c): Bias-variance trade-off and resampling techniques',
+ 'part-c-adding-lasso-for-the-franke-function'),
+ ('Part d): Paper and pencil part',
3,
None,
- 'part-c-bias-variance-trade-off-and-resampling-techniques'),
- ('Part d): Cross-validation as resampling techniques, adding '
+ 'part-d-paper-and-pencil-part'),
+ ('Part e): Bias-variance trade-off and resampling techniques',
+ 3,
+ None,
+ 'part-e-bias-variance-trade-off-and-resampling-techniques'),
+ ('Part f): Cross-validation as resampling techniques, adding '
'more complexity',
3,
None,
- 'part-d-cross-validation-as-resampling-techniques-adding-more-complexity'),
+ 'part-f-cross-validation-as-resampling-techniques-adding-more-complexity'),
('Part g): Analysis of real data',
3,
None,
@@ -120,10 +123,11 @@ MathJax.Hub.Config({
Regression analysis and resampling methods
Description of two-dimensional function
Part a) : Ordinary Least Square (OLS) on the Franke function
- Part b): Adding Ridge and Lasso Regression on the Franke function
- Part a): Paper and pencil part
- Part c): Bias-variance trade-off and resampling techniques
- Part d): Cross-validation as resampling techniques, adding more complexity
+ Part b): Adding Ridge regression for the Franke function
+ Part c): Adding Lasso for the Franke function
+ Part d): Paper and pencil part
+ Part e): Bias-variance trade-off and resampling techniques
+ Part f): Cross-validation as resampling techniques, adding more complexity
Part g): Analysis of real data
Background literature
Introduction to numerical projects
@@ -173,14 +177,14 @@ for example
we detail how to write a report. Furthermore, at
https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/
you can find examples of previous reports. How to write reports will
-also be discussed during lectures and at the various lab sessions.
+also be discussed during the various lab sessions.
A small recommendation when developing the codes here. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
-of week 35. A simple test, as discussed during the lectures the first
+of weeks 35 and 36. A simple test, as discussed during the lectures the first
two weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
@@ -212,12 +216,17 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}
\end{align*}
$$
-
The function will be defined for \( x,y\in [0,1] \). Our first step will
+
The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.
+
+Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y,
x^2, y^2, xy, \dots] \). We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
-technique. As discussed in the lectures for weeks 35 and 36,, we can
+technique.
+
+
+We can
use a uniform distribution to set up the arrays of values for \( x \) and
\( y \), or as in the example below just a set of fixed values for \( x \) and
\( y \) with a given step size. We will fit a function (for example a
@@ -352,26 +361,29 @@ approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
-Part b): Adding Ridge and Lasso Regression on the Franke function
+Part b): Adding Ridge regression for the Franke function
Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
-exercise.
+exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and
-analyze your results with those obtained in parts b-d). Study the
+analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on \( \lambda \).
+Part c): Adding Lasso for the Franke function
This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
-you can also use the functionalities of Scikit-Learn (recommended).
+you can also use the functionalities of Scikit-Learn (recommended). Keep in mind that the library Scikit-Learn excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
-model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the bootstrap resampling technique and an analysis of the mean squared error using cross-validation.
+model fits the data best.
-Part a): Paper and pencil part
+
+v
+Part d): Paper and pencil part
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
@@ -421,9 +433,11 @@ $$
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). .
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
-Part c): Bias-variance trade-off and resampling techniques
+Part e): Bias-variance trade-off and resampling techniques
-Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+We will only use the simpler ordinary least squares here.
+
With a code which does OLS and includes resampling techniques,
we will now discuss the bias-variance trade-off in the context of
@@ -446,7 +460,7 @@ dataset \( \mathcal{L} \) consisting of the data
\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
$$
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
@@ -503,12 +517,10 @@ of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the bootstrap resampling method.
You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff.
-Part d): Cross-validation as resampling techniques, adding more complexity
+Part f): Cross-validation as resampling techniques, adding more complexity
The aim here is to write your own code for another widely popular
-resampling technique, the so-called cross-validation method. Again,
-before you start with cross-validation approach, you should scale your
-data if you think this is needed.
+resampling technique, the so-called cross-validation method.
Implement the \( k \)-fold cross-validation algorithm (write your own
@@ -522,17 +534,19 @@ you got from your bootstrap code. Comment your results. Try \( 5-10 \)
folds. You can also compare your own cross-validation code with the
one provided by Scikit-Learn.
+
+In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
Part g): Analysis of real data
With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
-essentially repeat in this exercise what was done in exercises 1-5. However, we
+essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
https://earthexplorer.usgs.gov/,
-Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles
+Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles
In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want
diff --git a/doc/Projects/2023/Project1/html/Project1.html b/doc/Projects/2023/Project1/html/Project1.html
index 3e45c97e0..ef7099584 100644
--- a/doc/Projects/2023/Project1/html/Project1.html
+++ b/doc/Projects/2023/Project1/html/Project1.html
@@ -126,24 +126,27 @@ div.toc p,a {
3,
None,
'part-a-ordinary-least-square-ols-on-the-franke-function'),
- ('Part b): Adding Ridge and Lasso Regression on the Franke '
- 'function',
+ ('Part b): Adding Ridge regression for the Franke function',
3,
None,
- 'part-b-adding-ridge-and-lasso-regression-on-the-franke-function'),
- ('Part a): Paper and pencil part',
+ 'part-b-adding-ridge-regression-for-the-franke-function'),
+ ('Part c): Adding Lasso for the Franke function',
3,
None,
- 'part-a-paper-and-pencil-part'),
- ('Part c): Bias-variance trade-off and resampling techniques',
+ 'part-c-adding-lasso-for-the-franke-function'),
+ ('Part d): Paper and pencil part',
3,
None,
- 'part-c-bias-variance-trade-off-and-resampling-techniques'),
- ('Part d): Cross-validation as resampling techniques, adding '
+ 'part-d-paper-and-pencil-part'),
+ ('Part e): Bias-variance trade-off and resampling techniques',
+ 3,
+ None,
+ 'part-e-bias-variance-trade-off-and-resampling-techniques'),
+ ('Part f): Cross-validation as resampling techniques, adding '
'more complexity',
3,
None,
- 'part-d-cross-validation-as-resampling-techniques-adding-more-complexity'),
+ 'part-f-cross-validation-as-resampling-techniques-adding-more-complexity'),
('Part g): Analysis of real data',
3,
None,
@@ -210,14 +213,14 @@ for example
we detail how to write a report. Furthermore, at
https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/
you can find examples of previous reports. How to write reports will
-also be discussed during lectures and at the various lab sessions.
+also be discussed during the various lab sessions.
A small recommendation when developing the codes here. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
-of week 35. A simple test, as discussed during the lectures the first
+of weeks 35 and 36. A simple test, as discussed during the lectures the first
two weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
@@ -249,12 +252,17 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}
\end{align*}
$$
-
The function will be defined for \( x,y\in [0,1] \). Our first step will
+
The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.
+
+Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y,
x^2, y^2, xy, \dots] \). We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
-technique. As discussed in the lectures for weeks 35 and 36,, we can
+technique.
+
+
+We can
use a uniform distribution to set up the arrays of values for \( x \) and
\( y \), or as in the example below just a set of fixed values for \( x \) and
\( y \) with a given step size. We will fit a function (for example a
@@ -389,26 +397,29 @@ approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
-Part b): Adding Ridge and Lasso Regression on the Franke function
+Part b): Adding Ridge regression for the Franke function
Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
-exercise.
+exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and
-analyze your results with those obtained in parts b-d). Study the
+analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on \( \lambda \).
+Part c): Adding Lasso for the Franke function
This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
-you can also use the functionalities of Scikit-Learn (recommended).
+you can also use the functionalities of Scikit-Learn (recommended). Keep in mind that the library Scikit-Learn excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
-model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the bootstrap resampling technique and an analysis of the mean squared error using cross-validation.
+model fits the data best.
-Part a): Paper and pencil part
+
+v
+Part d): Paper and pencil part
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
@@ -458,9 +469,11 @@ $$
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). .
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
-Part c): Bias-variance trade-off and resampling techniques
+Part e): Bias-variance trade-off and resampling techniques
-Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+We will only use the simpler ordinary least squares here.
+
With a code which does OLS and includes resampling techniques,
we will now discuss the bias-variance trade-off in the context of
@@ -483,7 +496,7 @@ dataset \( \mathcal{L} \) consisting of the data
\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
$$
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
@@ -540,12 +553,10 @@ of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the bootstrap resampling method.
You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff.
-Part d): Cross-validation as resampling techniques, adding more complexity
+Part f): Cross-validation as resampling techniques, adding more complexity
The aim here is to write your own code for another widely popular
-resampling technique, the so-called cross-validation method. Again,
-before you start with cross-validation approach, you should scale your
-data if you think this is needed.
+resampling technique, the so-called cross-validation method.
Implement the \( k \)-fold cross-validation algorithm (write your own
@@ -559,17 +570,19 @@ you got from your bootstrap code. Comment your results. Try \( 5-10 \)
folds. You can also compare your own cross-validation code with the
one provided by Scikit-Learn.
+
+In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
Part g): Analysis of real data
With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
-essentially repeat in this exercise what was done in exercises 1-5. However, we
+essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
https://earthexplorer.usgs.gov/,
-Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles
+Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles
In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want
diff --git a/doc/Projects/2023/Project1/ipynb/Project1.ipynb b/doc/Projects/2023/Project1/ipynb/Project1.ipynb
index eb44608a1..c867ffa98 100644
--- a/doc/Projects/2023/Project1/ipynb/Project1.ipynb
+++ b/doc/Projects/2023/Project1/ipynb/Project1.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "9b31c43c",
+ "id": "1ceb1efb",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "de414429",
+ "id": "1f73d83b",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
- "id": "24422f72",
+ "id": "df2b7d33",
"metadata": {
"editable": true
},
@@ -44,13 +44,13 @@
"we detail how to write a report. Furthermore, at\n",
"\n",
"you can find examples of previous reports. How to write reports will\n",
- "also be discussed during lectures and at the various lab sessions.\n",
+ "also be discussed during the various lab sessions.\n",
"\n",
"**A small recommendation when developing the codes here**. Instead of\n",
"jumping on to the two-dimensional function described below, we\n",
"recommend to do the code development and testing with a simpler\n",
"one-dimensional function, similar to those discussed in the exercises\n",
- "of week 35. A simple test, as discussed during the lectures the first\n",
+ "of weeks 35 and 36. A simple test, as discussed during the lectures the first\n",
"two weeks is to set the design matrix equal to the identity\n",
"matrix. Then your model should give a mean square error which is exactly equal to zero.\n",
"When you are sure that your codes function well, you can then replace\n",
@@ -63,7 +63,7 @@
},
{
"cell_type": "markdown",
- "id": "2f57baf4",
+ "id": "83787bae",
"metadata": {
"editable": true
},
@@ -85,7 +85,7 @@
},
{
"cell_type": "markdown",
- "id": "1cd79adf",
+ "id": "d7ee2bfd",
"metadata": {
"editable": true
},
@@ -100,17 +100,21 @@
},
{
"cell_type": "markdown",
- "id": "35c04be8",
+ "id": "958a455e",
"metadata": {
"editable": true
},
"source": [
- "The function will be defined for $x,y\\in [0,1]$. Our first step will\n",
+ "The function will be defined for $x,y\\in [0,1]$. In a sense, our data are thus scaled to a particular domain for the input values.\n",
+ "\n",
+ "Our first step will\n",
"be to perform an OLS regression analysis of this function, trying out\n",
"a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,\n",
"x^2, y^2, xy, \\dots]$. We will also include bootstrap first as a\n",
"resampling technique. After that we will include the cross-validation\n",
- "technique. As discussed in the lectures for weeks 35 and 36,, we can\n",
+ "technique.\n",
+ "\n",
+ "We can\n",
"use a uniform distribution to set up the arrays of values for $x$ and\n",
"$y$, or as in the example below just a set of fixed values for $x$ and\n",
"$y$ with a given step size. We will fit a function (for example a\n",
@@ -129,7 +133,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "d531f23c",
+ "id": "3deb0a35",
"metadata": {
"collapsed": false,
"editable": true
@@ -181,7 +185,7 @@
},
{
"cell_type": "markdown",
- "id": "3117b409",
+ "id": "742ae4b5",
"metadata": {
"editable": true
},
@@ -203,7 +207,7 @@
},
{
"cell_type": "markdown",
- "id": "b10e86f2",
+ "id": "e49ae3a8",
"metadata": {
"editable": true
},
@@ -216,7 +220,7 @@
},
{
"cell_type": "markdown",
- "id": "71459fea",
+ "id": "091e8f0d",
"metadata": {
"editable": true
},
@@ -228,7 +232,7 @@
},
{
"cell_type": "markdown",
- "id": "f5687714",
+ "id": "3fad86e6",
"metadata": {
"editable": true
},
@@ -240,7 +244,7 @@
},
{
"cell_type": "markdown",
- "id": "b7ddfef4",
+ "id": "d281bfb0",
"metadata": {
"editable": true
},
@@ -250,7 +254,7 @@
},
{
"cell_type": "markdown",
- "id": "5ffbd03c",
+ "id": "af6fb771",
"metadata": {
"editable": true
},
@@ -262,7 +266,7 @@
},
{
"cell_type": "markdown",
- "id": "04963148",
+ "id": "0d89eb4e",
"metadata": {
"editable": true
},
@@ -291,37 +295,49 @@
},
{
"cell_type": "markdown",
- "id": "d17e5544",
+ "id": "e1c30216",
"metadata": {
"editable": true
},
"source": [
- "### Part b): Adding Ridge and Lasso Regression on the Franke function\n",
+ "### Part b): Adding Ridge regression for the Franke function\n",
"\n",
"Write your own code for the Ridge method, either using matrix\n",
"inversion or the singular value decomposition as done in the previous\n",
- "exercise.\n",
+ "exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.\n",
"\n",
"Perform the same analysis as you did in the previous exercise but now for different values of $\\lambda$. Compare and\n",
- "analyze your results with those obtained in parts b-d). Study the\n",
- "dependence on $\\lambda$.\n",
- "\n",
- "This exercise is essentially a repeat of the previous two ones, but now\n",
- "with Lasso regression. Write either your own code (difficult and optional) or, in this case,\n",
- "you can also use the functionalities of **Scikit-Learn** (recommended). \n",
- "Give a\n",
- "critical discussion of the three methods and a judgement of which\n",
- "model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the **bootstrap** resampling technique and an analysis of the mean squared error using cross-validation."
+ "analyze your results with those obtained in part a) with the ordinary least squares method. Study the\n",
+ "dependence on $\\lambda$."
]
},
{
"cell_type": "markdown",
- "id": "fbd4c145",
+ "id": "b121745d",
"metadata": {
"editable": true
},
"source": [
- "### Part a): Paper and pencil part\n",
+ "### Part c): Adding Lasso for the Franke function\n",
+ "\n",
+ "This exercise is essentially a repeat of the previous two ones, but now\n",
+ "with Lasso regression. Write either your own code (difficult and optional) or, in this case,\n",
+ "you can also use the functionalities of **Scikit-Learn** (recommended). Keep in mind that the library **Scikit-Learn** excludes the intercept by default. \n",
+ "Give a\n",
+ "critical discussion of the three methods and a judgement of which\n",
+ "model fits the data best.\n",
+ "\n",
+ "v"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "56d96043",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "### Part d): Paper and pencil part\n",
"\n",
"This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)).\n",
"\n",
@@ -332,7 +348,7 @@
},
{
"cell_type": "markdown",
- "id": "ce635f60",
+ "id": "e64c004b",
"metadata": {
"editable": true
},
@@ -344,7 +360,7 @@
},
{
"cell_type": "markdown",
- "id": "d55bb605",
+ "id": "6f17d2da",
"metadata": {
"editable": true
},
@@ -355,7 +371,7 @@
},
{
"cell_type": "markdown",
- "id": "1a43accc",
+ "id": "fc4fe2fb",
"metadata": {
"editable": true
},
@@ -367,7 +383,7 @@
},
{
"cell_type": "markdown",
- "id": "8a708d42",
+ "id": "93babb22",
"metadata": {
"editable": true
},
@@ -379,7 +395,7 @@
},
{
"cell_type": "markdown",
- "id": "a5d55b46",
+ "id": "4b4061ba",
"metadata": {
"editable": true
},
@@ -391,7 +407,7 @@
},
{
"cell_type": "markdown",
- "id": "19e93313",
+ "id": "1ca5e57a",
"metadata": {
"editable": true
},
@@ -402,7 +418,7 @@
},
{
"cell_type": "markdown",
- "id": "bb735793",
+ "id": "da25c4ba",
"metadata": {
"editable": true
},
@@ -414,7 +430,7 @@
},
{
"cell_type": "markdown",
- "id": "ee65e390",
+ "id": "10ce6cfd",
"metadata": {
"editable": true
},
@@ -427,7 +443,7 @@
},
{
"cell_type": "markdown",
- "id": "b29b4187",
+ "id": "8d3f716a",
"metadata": {
"editable": true
},
@@ -439,7 +455,7 @@
},
{
"cell_type": "markdown",
- "id": "f8adffa4",
+ "id": "e30af070",
"metadata": {
"editable": true
},
@@ -449,7 +465,7 @@
},
{
"cell_type": "markdown",
- "id": "1f56f45d",
+ "id": "1bc3edc7",
"metadata": {
"editable": true
},
@@ -461,7 +477,7 @@
},
{
"cell_type": "markdown",
- "id": "74e687b8",
+ "id": "050e1f04",
"metadata": {
"editable": true
},
@@ -472,14 +488,15 @@
},
{
"cell_type": "markdown",
- "id": "b0b7d7bb",
+ "id": "52486468",
"metadata": {
"editable": true
},
"source": [
- "### Part c): Bias-variance trade-off and resampling techniques\n",
+ "### Part e): Bias-variance trade-off and resampling techniques\n",
"\n",
"Our aim here is to study the bias-variance trade-off by implementing the **bootstrap** resampling technique.\n",
+ "**We will only use the simpler ordinary least squares here**.\n",
"\n",
"With a code which does OLS and includes resampling techniques, \n",
"we will now discuss the bias-variance trade-off in the context of\n",
@@ -499,12 +516,12 @@
"dataset $\\mathcal{L}$ consisting of the data\n",
"$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n",
"\n",
- "As in part a), we assume that the true data is generated from a noisy model"
+ "As in part d), we assume that the true data is generated from a noisy model"
]
},
{
"cell_type": "markdown",
- "id": "6f7509d8",
+ "id": "008e92a6",
"metadata": {
"editable": true
},
@@ -516,7 +533,7 @@
},
{
"cell_type": "markdown",
- "id": "8d06ec98",
+ "id": "123f65c5",
"metadata": {
"editable": true
},
@@ -535,7 +552,7 @@
},
{
"cell_type": "markdown",
- "id": "9e493a94",
+ "id": "a4398be8",
"metadata": {
"editable": true
},
@@ -547,7 +564,7 @@
},
{
"cell_type": "markdown",
- "id": "b6b7b71c",
+ "id": "3e5bee01",
"metadata": {
"editable": true
},
@@ -561,7 +578,7 @@
},
{
"cell_type": "markdown",
- "id": "eadda792",
+ "id": "e35fdf55",
"metadata": {
"editable": true
},
@@ -573,7 +590,7 @@
},
{
"cell_type": "markdown",
- "id": "cef4fa19",
+ "id": "ac0994d8",
"metadata": {
"editable": true
},
@@ -583,7 +600,7 @@
},
{
"cell_type": "markdown",
- "id": "38cc8dbb",
+ "id": "65ba61ff",
"metadata": {
"editable": true
},
@@ -595,7 +612,7 @@
},
{
"cell_type": "markdown",
- "id": "a9dcfd8b",
+ "id": "dd4e4536",
"metadata": {
"editable": true
},
@@ -605,7 +622,7 @@
},
{
"cell_type": "markdown",
- "id": "41198b5e",
+ "id": "7730a37d",
"metadata": {
"editable": true
},
@@ -617,7 +634,7 @@
},
{
"cell_type": "markdown",
- "id": "a252009f",
+ "id": "8cefeb53",
"metadata": {
"editable": true
},
@@ -636,17 +653,15 @@
},
{
"cell_type": "markdown",
- "id": "00854ec4",
+ "id": "8ec208c4",
"metadata": {
"editable": true
},
"source": [
- "### Part d): Cross-validation as resampling techniques, adding more complexity\n",
+ "### Part f): Cross-validation as resampling techniques, adding more complexity\n",
"\n",
"The aim here is to write your own code for another widely popular\n",
- "resampling technique, the so-called cross-validation method. Again,\n",
- "before you start with cross-validation approach, you should scale your\n",
- "data if you think this is needed.\n",
+ "resampling technique, the so-called cross-validation method. \n",
"\n",
"Implement the $k$-fold cross-validation algorithm (write your own\n",
"code) and evaluate again the MSE function resulting\n",
@@ -656,12 +671,14 @@
"Compare the MSE you get from your cross-validation code with the one\n",
"you got from your **bootstrap** code. Comment your results. Try $5-10$\n",
"folds. You can also compare your own cross-validation code with the\n",
- "one provided by **Scikit-Learn**."
+ "one provided by **Scikit-Learn**.\n",
+ "\n",
+ "In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression."
]
},
{
"cell_type": "markdown",
- "id": "0603ed7b",
+ "id": "b142cd65",
"metadata": {
"editable": true
},
@@ -670,12 +687,12 @@
"\n",
"With our codes functioning and having been tested properly on a\n",
"simpler function we are now ready to look at real data. We will\n",
- "essentially repeat in this exercise what was done in exercises 1-5. However, we\n",
+ "essentially repeat in this exercise what was done in exercises a-f. However, we\n",
"need first to download the data and prepare properly the inputs to our\n",
"codes. We are going to download digital terrain data from the website\n",
",\n",
"\n",
- "Or, if you prefer, we have placed selected datafiles at \n",
+ "Or, if you prefer, we have placed selected datafiles at \n",
"\n",
"In order to obtain data for a specific region, you need to register as\n",
"a user (free) at this website and then decide upon which area you want\n",
@@ -689,7 +706,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "293624f7",
+ "id": "fe878605",
"metadata": {
"collapsed": false,
"editable": true
@@ -701,7 +718,7 @@
},
{
"cell_type": "markdown",
- "id": "2eba6a2c",
+ "id": "10b84fbb",
"metadata": {
"editable": true
},
@@ -713,7 +730,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "a7f37cdb",
+ "id": "e845fd10",
"metadata": {
"collapsed": false,
"editable": true
@@ -739,7 +756,7 @@
},
{
"cell_type": "markdown",
- "id": "74af5abe",
+ "id": "0745b3d5",
"metadata": {
"editable": true
},
@@ -764,7 +781,7 @@
},
{
"cell_type": "markdown",
- "id": "5ad507ad",
+ "id": "2ce7a62f",
"metadata": {
"editable": true
},
@@ -778,7 +795,7 @@
},
{
"cell_type": "markdown",
- "id": "194f7c51",
+ "id": "34b72e9a",
"metadata": {
"editable": true
},
@@ -808,7 +825,7 @@
},
{
"cell_type": "markdown",
- "id": "a1cf04b9",
+ "id": "d0f3a4fc",
"metadata": {
"editable": true
},
@@ -830,7 +847,7 @@
},
{
"cell_type": "markdown",
- "id": "421f2a0c",
+ "id": "436bcd57",
"metadata": {
"editable": true
},
diff --git a/doc/Projects/2023/Project1/ipynb/ipynb-Project1-src.tar.gz b/doc/Projects/2023/Project1/ipynb/ipynb-Project1-src.tar.gz
index b06940a4c..ca520b06a 100644
Binary files a/doc/Projects/2023/Project1/ipynb/ipynb-Project1-src.tar.gz and b/doc/Projects/2023/Project1/ipynb/ipynb-Project1-src.tar.gz differ
diff --git a/doc/Projects/2023/Project1/pdf/Project1.p.tex b/doc/Projects/2023/Project1/pdf/Project1.p.tex
index 0f998ea00..da544d658 100644
--- a/doc/Projects/2023/Project1/pdf/Project1.p.tex
+++ b/doc/Projects/2023/Project1/pdf/Project1.p.tex
@@ -151,13 +151,13 @@ for example
we detail how to write a report. Furthermore, at
\href{{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/}}{\nolinkurl{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/}}
you can find examples of previous reports. How to write reports will
-also be discussed during lectures and at the various lab sessions.
+also be discussed during the various lab sessions.
\textbf{A small recommendation when developing the codes here}. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
-of week 35. A simple test, as discussed during the lectures the first
+of weeks 35 and 36. A simple test, as discussed during the lectures the first
two weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
@@ -184,12 +184,16 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
\end{align*}
-The function will be defined for $x,y\in [0,1]$. Our first step will
+The function will be defined for $x,y\in [0,1]$. In a sense, our data are thus scaled to a particular domain for the input values.
+
+Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,
x^2, y^2, xy, \dots]$. We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
-technique. As discussed in the lectures for weeks 35 and 36,, we can
+technique.
+
+We can
use a uniform distribution to set up the arrays of values for $x$ and
$y$, or as in the example below just a set of fixed values for $x$ and
$y$ with a given step size. We will fit a function (for example a
@@ -341,23 +345,26 @@ approximately $2/3$ to $4/5$ of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
-\paragraph{Part b): Adding Ridge and Lasso Regression on the Franke function.}
+\paragraph{Part b): Adding Ridge regression for the Franke function.}
Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
-exercise.
+exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of $\lambda$. Compare and
-analyze your results with those obtained in parts b-d). Study the
+analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on $\lambda$.
+\paragraph{Part c): Adding Lasso for the Franke function.}
This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
-you can also use the functionalities of \textbf{Scikit-Learn} (recommended).
+you can also use the functionalities of \textbf{Scikit-Learn} (recommended). Keep in mind that the library \textbf{Scikit-Learn} excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
-model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the \textbf{bootstrap} resampling technique and an analysis of the mean squared error using cross-validation.
+model fits the data best.
-\paragraph{Part a): Paper and pencil part.}
+v
+
+\paragraph{Part d): Paper and pencil part.}
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}).
The assumption we have made is
@@ -398,8 +405,9 @@ Show finally that the variance of $\bm{\beta}$ is
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$. .
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
-\paragraph{Part c): Bias-variance trade-off and resampling techniques.}
+\paragraph{Part e): Bias-variance trade-off and resampling techniques.}
Our aim here is to study the bias-variance trade-off by implementing the \textbf{bootstrap} resampling technique.
+\textbf{We will only use the simpler ordinary least squares here}.
With a code which does OLS and includes resampling techniques,
we will now discuss the bias-variance trade-off in the context of
@@ -419,7 +427,7 @@ Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
\[
\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}.
@@ -466,11 +474,9 @@ of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method.
You can follow the code example in the jupyter-book at \href{{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff}}{\nolinkurl{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html\#the-bias-variance-tradeoff}}.
-\paragraph{Part d): Cross-validation as resampling techniques, adding more complexity.}
+\paragraph{Part f): Cross-validation as resampling techniques, adding more complexity.}
The aim here is to write your own code for another widely popular
-resampling technique, the so-called cross-validation method. Again,
-before you start with cross-validation approach, you should scale your
-data if you think this is needed.
+resampling technique, the so-called cross-validation method.
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
@@ -482,15 +488,17 @@ you got from your \textbf{bootstrap} code. Comment your results. Try $5-10$
folds. You can also compare your own cross-validation code with the
one provided by \textbf{Scikit-Learn}.
+In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
+
\paragraph{Part g): Analysis of real data.}
With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
-essentially repeat in this exercise what was done in exercises 1-5. However, we
+essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
\href{{https://earthexplorer.usgs.gov/}}{\nolinkurl{https://earthexplorer.usgs.gov/}},
-Or, if you prefer, we have placed selected datafiles at \href{{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles}}{\nolinkurl{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles}}
+Or, if you prefer, we have placed selected datafiles at \href{{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles}}{\nolinkurl{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles}}
In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want
diff --git a/doc/Projects/2023/Project1/pdf/Project1.pdf b/doc/Projects/2023/Project1/pdf/Project1.pdf
index 71e34f984..c0705095b 100644
Binary files a/doc/Projects/2023/Project1/pdf/Project1.pdf and b/doc/Projects/2023/Project1/pdf/Project1.pdf differ
diff --git a/doc/Projects/2023/Project1/pdf/Project1.tex b/doc/Projects/2023/Project1/pdf/Project1.tex
index e602f817e..b495b5aeb 100644
--- a/doc/Projects/2023/Project1/pdf/Project1.tex
+++ b/doc/Projects/2023/Project1/pdf/Project1.tex
@@ -121,13 +121,13 @@ for example
we detail how to write a report. Furthermore, at
\href{{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/}}{\nolinkurl{https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/}}
you can find examples of previous reports. How to write reports will
-also be discussed during lectures and at the various lab sessions.
+also be discussed during the various lab sessions.
\textbf{A small recommendation when developing the codes here}. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
-of week 35. A simple test, as discussed during the lectures the first
+of weeks 35 and 36. A simple test, as discussed during the lectures the first
two weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
@@ -154,12 +154,16 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
\end{align*}
-The function will be defined for $x,y\in [0,1]$. Our first step will
+The function will be defined for $x,y\in [0,1]$. In a sense, our data are thus scaled to a particular domain for the input values.
+
+Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,
x^2, y^2, xy, \dots]$. We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
-technique. As discussed in the lectures for weeks 35 and 36,, we can
+technique.
+
+We can
use a uniform distribution to set up the arrays of values for $x$ and
$y$, or as in the example below just a set of fixed values for $x$ and
$y$ with a given step size. We will fit a function (for example a
@@ -311,23 +315,26 @@ approximately $2/3$ to $4/5$ of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
-\paragraph{Part b): Adding Ridge and Lasso Regression on the Franke function.}
+\paragraph{Part b): Adding Ridge regression for the Franke function.}
Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
-exercise.
+exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of $\lambda$. Compare and
-analyze your results with those obtained in parts b-d). Study the
+analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on $\lambda$.
+\paragraph{Part c): Adding Lasso for the Franke function.}
This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
-you can also use the functionalities of \textbf{Scikit-Learn} (recommended).
+you can also use the functionalities of \textbf{Scikit-Learn} (recommended). Keep in mind that the library \textbf{Scikit-Learn} excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
-model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the \textbf{bootstrap} resampling technique and an analysis of the mean squared error using cross-validation.
+model fits the data best.
-\paragraph{Part a): Paper and pencil part.}
+v
+
+\paragraph{Part d): Paper and pencil part.}
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}).
The assumption we have made is
@@ -368,8 +375,9 @@ Show finally that the variance of $\bm{\beta}$ is
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$. .
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
-\paragraph{Part c): Bias-variance trade-off and resampling techniques.}
+\paragraph{Part e): Bias-variance trade-off and resampling techniques.}
Our aim here is to study the bias-variance trade-off by implementing the \textbf{bootstrap} resampling technique.
+\textbf{We will only use the simpler ordinary least squares here}.
With a code which does OLS and includes resampling techniques,
we will now discuss the bias-variance trade-off in the context of
@@ -389,7 +397,7 @@ Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
\[
\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}.
@@ -436,11 +444,9 @@ of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method.
You can follow the code example in the jupyter-book at \href{{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff}}{\nolinkurl{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html\#the-bias-variance-tradeoff}}.
-\paragraph{Part d): Cross-validation as resampling techniques, adding more complexity.}
+\paragraph{Part f): Cross-validation as resampling techniques, adding more complexity.}
The aim here is to write your own code for another widely popular
-resampling technique, the so-called cross-validation method. Again,
-before you start with cross-validation approach, you should scale your
-data if you think this is needed.
+resampling technique, the so-called cross-validation method.
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
@@ -452,15 +458,17 @@ you got from your \textbf{bootstrap} code. Comment your results. Try $5-10$
folds. You can also compare your own cross-validation code with the
one provided by \textbf{Scikit-Learn}.
+In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
+
\paragraph{Part g): Analysis of real data.}
With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
-essentially repeat in this exercise what was done in exercises 1-5. However, we
+essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
\href{{https://earthexplorer.usgs.gov/}}{\nolinkurl{https://earthexplorer.usgs.gov/}},
-Or, if you prefer, we have placed selected datafiles at \href{{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles}}{\nolinkurl{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles}}
+Or, if you prefer, we have placed selected datafiles at \href{{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles}}{\nolinkurl{https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles}}
In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want
diff --git a/doc/src/Projects/2023/Project1/Project1.do.txt b/doc/src/Projects/2023/Project1/Project1.do.txt
index ba711ea43..cc9a9b3af 100644
--- a/doc/src/Projects/2023/Project1/Project1.do.txt
+++ b/doc/src/Projects/2023/Project1/Project1.do.txt
@@ -18,14 +18,14 @@ URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/Eva
we detail how to write a report. Furthermore, at
URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/"
you can find examples of previous reports. How to write reports will
-also be discussed during lectures and at the various lab sessions.
+also be discussed during the various lab sessions.
_A small recommendation when developing the codes here_. Instead of
jumping on to the two-dimensional function described below, we
recommend to do the code development and testing with a simpler
one-dimensional function, similar to those discussed in the exercises
-of week 35. A simple test, as discussed during the lectures the first
+of weeks 35 and 36. A simple test, as discussed during the lectures the first
two weeks is to set the design matrix equal to the identity
matrix. Then your model should give a mean square error which is exactly equal to zero.
When you are sure that your codes function well, you can then replace
@@ -57,12 +57,16 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}
\end{align*}
!et
-The function will be defined for $x,y\in [0,1]$. Our first step will
+The function will be defined for $x,y\in [0,1]$. In a sense, our data are thus scaled to a particular domain for the input values.
+
+Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,
x^2, y^2, xy, \dots]$. We will also include bootstrap first as a
resampling technique. After that we will include the cross-validation
-technique. As discussed in the lectures for weeks 35 and 36,, we can
+technique.
+
+We can
use a uniform distribution to set up the arrays of values for $x$ and
$y$, or as in the example below just a set of fixed values for $x$ and
$y$ with a given step size. We will fit a function (for example a
@@ -184,25 +188,28 @@ You can easily reuse the solutions to your exercises from week 35 and week 36.
See also the lecture slides from week 35 and week 36.
-=== Part b): Adding Ridge and Lasso Regression on the Franke function ===
+=== Part b): Adding Ridge regression for the Franke function ===
Write your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done in the previous
-exercise.
+exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of $\lambda$. Compare and
-analyze your results with those obtained in parts b-d). Study the
+analyze your results with those obtained in part a) with the ordinary least squares method. Study the
dependence on $\lambda$.
+=== Part c): Adding Lasso for the Franke function ===
+
This exercise is essentially a repeat of the previous two ones, but now
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
-you can also use the functionalities of _Scikit-Learn_ (recommended).
+you can also use the functionalities of _Scikit-Learn_ (recommended). Keep in mind that the library _Scikit-Learn_ excludes the intercept by default.
Give a
critical discussion of the three methods and a judgement of which
-model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the _bootstrap_ resampling technique and an analysis of the mean squared error using cross-validation.
+model fits the data best.
+v
-=== Part a): Paper and pencil part ===
+=== Part d): Paper and pencil part ===
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570").
@@ -262,9 +269,11 @@ A given parameter $\beta_j$ is given by the diagonal matrix element of the above
-=== Part c): Bias-variance trade-off and resampling techniques ===
+=== Part e): Bias-variance trade-off and resampling techniques ===
Our aim here is to study the bias-variance trade-off by implementing the _bootstrap_ resampling technique.
+_We will only use the simpler ordinary least squares here_.
+
With a code which does OLS and includes resampling techniques,
we will now discuss the bias-variance trade-off in the context of
@@ -284,7 +293,7 @@ Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
!bt
\[
@@ -342,13 +351,12 @@ of data points, and possibly also your training and test data using the _bootstr
You can follow the code example in the jupyter-book at URL:"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff".
-=== Part d): Cross-validation as resampling techniques, adding more complexity ===
+
+=== Part f): Cross-validation as resampling techniques, adding more complexity ===
The aim here is to write your own code for another widely popular
-resampling technique, the so-called cross-validation method. Again,
-before you start with cross-validation approach, you should scale your
-data if you think this is needed.
+resampling technique, the so-called cross-validation method.
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
@@ -360,18 +368,19 @@ you got from your _bootstrap_ code. Comment your results. Try $5-10$
folds. You can also compare your own cross-validation code with the
one provided by _Scikit-Learn_.
+In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
=== Part g): Analysis of real data ===
With our codes functioning and having been tested properly on a
simpler function we are now ready to look at real data. We will
-essentially repeat in this exercise what was done in exercises 1-5. However, we
+essentially repeat in this exercise what was done in exercises a-f. However, we
need first to download the data and prepare properly the inputs to our
codes. We are going to download digital terrain data from the website
URL:"https://earthexplorer.usgs.gov/",
-Or, if you prefer, we have placed selected datafiles at URL:"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles"
+Or, if you prefer, we have placed selected datafiles at URL:"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles"
In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want