From 2a445d0f022b9ce8e4dd86c45e39f09063193a24 Mon Sep 17 00:00:00 2001
From: Morten Hjorth-Jensen
A small recommendation when developing the codes here. Instead of jumping on to the two-dimensional function described below, we recommend to do the code development and testing with a simpler one-dimensional function, similar to those discussed in the exercises -of week 35. A simple test, as discussed during the lectures the first +of weeks 35 and 36. A simple test, as discussed during the lectures the first two weeks is to set the design matrix equal to the identity matrix. Then your model should give a mean square error which is exactly equal to zero. When you are sure that your codes function well, you can then replace @@ -212,12 +216,17 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)} \end{align*} $$ -
The function will be defined for \( x,y\in [0,1] \). Our first step will +
The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.
+ +Our first step will be to perform an OLS regression analysis of this function, trying out a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, x^2, y^2, xy, \dots] \). We will also include bootstrap first as a resampling technique. After that we will include the cross-validation -technique. As discussed in the lectures for weeks 35 and 36,, we can +technique. +
+ +We can use a uniform distribution to set up the arrays of values for \( x \) and \( y \), or as in the example below just a set of fixed values for \( x \) and \( y \) with a given step size. We will fit a function (for example a @@ -352,26 +361,29 @@ approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36. See also the lecture slides from week 35 and week 36.
-Write your own code for the Ridge method, either using matrix inversion or the singular value decomposition as done in the previous -exercise. +exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and -analyze your results with those obtained in parts b-d). Study the +analyze your results with those obtained in part a) with the ordinary least squares method. Study the dependence on \( \lambda \).
+This exercise is essentially a repeat of the previous two ones, but now with Lasso regression. Write either your own code (difficult and optional) or, in this case, -you can also use the functionalities of Scikit-Learn (recommended). +you can also use the functionalities of Scikit-Learn (recommended). Keep in mind that the library Scikit-Learn excludes the intercept by default. Give a critical discussion of the three methods and a judgement of which -model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the bootstrap resampling technique and an analysis of the mean squared error using cross-validation. +model fits the data best.
-v
+This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
@@ -421,9 +433,11 @@ $$We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). . A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
-Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique. +We will only use the simpler ordinary least squares here. +
With a code which does OLS and includes resampling techniques, we will now discuss the bias-variance trade-off in the context of @@ -446,7 +460,7 @@ dataset \( \mathcal{L} \) consisting of the data \( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
$$ \boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}. @@ -503,12 +517,10 @@ of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method. You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff. -The aim here is to write your own code for another widely popular -resampling technique, the so-called cross-validation method. Again, -before you start with cross-validation approach, you should scale your -data if you think this is needed. +resampling technique, the so-called cross-validation method.
Implement the \( k \)-fold cross-validation algorithm (write your own @@ -522,17 +534,19 @@ you got from your bootstrap code. Comment your results. Try \( 5-10 \) folds. You can also compare your own cross-validation code with the one provided by Scikit-Learn.
+ +In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
With our codes functioning and having been tested properly on a simpler function we are now ready to look at real data. We will -essentially repeat in this exercise what was done in exercises 1-5. However, we +essentially repeat in this exercise what was done in exercises a-f. However, we need first to download the data and prepare properly the inputs to our codes. We are going to download digital terrain data from the website https://earthexplorer.usgs.gov/,
-Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles
+Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles
In order to obtain data for a specific region, you need to register as a user (free) at this website and then decide upon which area you want diff --git a/doc/Projects/2023/Project1/html/Project1-bs.html b/doc/Projects/2023/Project1/html/Project1-bs.html index e4b893262..3a8cec6ec 100644 --- a/doc/Projects/2023/Project1/html/Project1-bs.html +++ b/doc/Projects/2023/Project1/html/Project1-bs.html @@ -48,24 +48,27 @@ doconce format html Project1.do.txt --html_style=bootstrap --pygments_html_style 3, None, 'part-a-ordinary-least-square-ols-on-the-franke-function'), - ('Part b): Adding Ridge and Lasso Regression on the Franke ' - 'function', + ('Part b): Adding Ridge regression for the Franke function', 3, None, - 'part-b-adding-ridge-and-lasso-regression-on-the-franke-function'), - ('Part a): Paper and pencil part', + 'part-b-adding-ridge-regression-for-the-franke-function'), + ('Part c): Adding Lasso for the Franke function', 3, None, - 'part-a-paper-and-pencil-part'), - ('Part c): Bias-variance trade-off and resampling techniques', + 'part-c-adding-lasso-for-the-franke-function'), + ('Part d): Paper and pencil part', 3, None, - 'part-c-bias-variance-trade-off-and-resampling-techniques'), - ('Part d): Cross-validation as resampling techniques, adding ' + 'part-d-paper-and-pencil-part'), + ('Part e): Bias-variance trade-off and resampling techniques', + 3, + None, + 'part-e-bias-variance-trade-off-and-resampling-techniques'), + ('Part f): Cross-validation as resampling techniques, adding ' 'more complexity', 3, None, - 'part-d-cross-validation-as-resampling-techniques-adding-more-complexity'), + 'part-f-cross-validation-as-resampling-techniques-adding-more-complexity'), ('Part g): Analysis of real data', 3, None, @@ -120,10 +123,11 @@ MathJax.Hub.Config({
A small recommendation when developing the codes here. Instead of jumping on to the two-dimensional function described below, we recommend to do the code development and testing with a simpler one-dimensional function, similar to those discussed in the exercises -of week 35. A simple test, as discussed during the lectures the first +of weeks 35 and 36. A simple test, as discussed during the lectures the first two weeks is to set the design matrix equal to the identity matrix. Then your model should give a mean square error which is exactly equal to zero. When you are sure that your codes function well, you can then replace @@ -212,12 +216,17 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)} \end{align*} $$ -
The function will be defined for \( x,y\in [0,1] \). Our first step will +
The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.
+ +Our first step will be to perform an OLS regression analysis of this function, trying out a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, x^2, y^2, xy, \dots] \). We will also include bootstrap first as a resampling technique. After that we will include the cross-validation -technique. As discussed in the lectures for weeks 35 and 36,, we can +technique. +
+ +We can use a uniform distribution to set up the arrays of values for \( x \) and \( y \), or as in the example below just a set of fixed values for \( x \) and \( y \) with a given step size. We will fit a function (for example a @@ -352,26 +361,29 @@ approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36. See also the lecture slides from week 35 and week 36.
-Write your own code for the Ridge method, either using matrix inversion or the singular value decomposition as done in the previous -exercise. +exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and -analyze your results with those obtained in parts b-d). Study the +analyze your results with those obtained in part a) with the ordinary least squares method. Study the dependence on \( \lambda \).
+This exercise is essentially a repeat of the previous two ones, but now with Lasso regression. Write either your own code (difficult and optional) or, in this case, -you can also use the functionalities of Scikit-Learn (recommended). +you can also use the functionalities of Scikit-Learn (recommended). Keep in mind that the library Scikit-Learn excludes the intercept by default. Give a critical discussion of the three methods and a judgement of which -model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the bootstrap resampling technique and an analysis of the mean squared error using cross-validation. +model fits the data best.
-v
+This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
@@ -421,9 +433,11 @@ $$We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). . A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
-Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique. +We will only use the simpler ordinary least squares here. +
With a code which does OLS and includes resampling techniques, we will now discuss the bias-variance trade-off in the context of @@ -446,7 +460,7 @@ dataset \( \mathcal{L} \) consisting of the data \( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
$$ \boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}. @@ -503,12 +517,10 @@ of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method. You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff. -The aim here is to write your own code for another widely popular -resampling technique, the so-called cross-validation method. Again, -before you start with cross-validation approach, you should scale your -data if you think this is needed. +resampling technique, the so-called cross-validation method.
Implement the \( k \)-fold cross-validation algorithm (write your own @@ -522,17 +534,19 @@ you got from your bootstrap code. Comment your results. Try \( 5-10 \) folds. You can also compare your own cross-validation code with the one provided by Scikit-Learn.
+ +In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
With our codes functioning and having been tested properly on a simpler function we are now ready to look at real data. We will -essentially repeat in this exercise what was done in exercises 1-5. However, we +essentially repeat in this exercise what was done in exercises a-f. However, we need first to download the data and prepare properly the inputs to our codes. We are going to download digital terrain data from the website https://earthexplorer.usgs.gov/,
-Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles
+Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles
In order to obtain data for a specific region, you need to register as a user (free) at this website and then decide upon which area you want diff --git a/doc/Projects/2023/Project1/html/Project1.html b/doc/Projects/2023/Project1/html/Project1.html index 3e45c97e0..ef7099584 100644 --- a/doc/Projects/2023/Project1/html/Project1.html +++ b/doc/Projects/2023/Project1/html/Project1.html @@ -126,24 +126,27 @@ div.toc p,a { 3, None, 'part-a-ordinary-least-square-ols-on-the-franke-function'), - ('Part b): Adding Ridge and Lasso Regression on the Franke ' - 'function', + ('Part b): Adding Ridge regression for the Franke function', 3, None, - 'part-b-adding-ridge-and-lasso-regression-on-the-franke-function'), - ('Part a): Paper and pencil part', + 'part-b-adding-ridge-regression-for-the-franke-function'), + ('Part c): Adding Lasso for the Franke function', 3, None, - 'part-a-paper-and-pencil-part'), - ('Part c): Bias-variance trade-off and resampling techniques', + 'part-c-adding-lasso-for-the-franke-function'), + ('Part d): Paper and pencil part', 3, None, - 'part-c-bias-variance-trade-off-and-resampling-techniques'), - ('Part d): Cross-validation as resampling techniques, adding ' + 'part-d-paper-and-pencil-part'), + ('Part e): Bias-variance trade-off and resampling techniques', + 3, + None, + 'part-e-bias-variance-trade-off-and-resampling-techniques'), + ('Part f): Cross-validation as resampling techniques, adding ' 'more complexity', 3, None, - 'part-d-cross-validation-as-resampling-techniques-adding-more-complexity'), + 'part-f-cross-validation-as-resampling-techniques-adding-more-complexity'), ('Part g): Analysis of real data', 3, None, @@ -210,14 +213,14 @@ for example we detail how to write a report. Furthermore, at https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/ReportExample/ you can find examples of previous reports. How to write reports will -also be discussed during lectures and at the various lab sessions. +also be discussed during the various lab sessions.
A small recommendation when developing the codes here. Instead of jumping on to the two-dimensional function described below, we recommend to do the code development and testing with a simpler one-dimensional function, similar to those discussed in the exercises -of week 35. A simple test, as discussed during the lectures the first +of weeks 35 and 36. A simple test, as discussed during the lectures the first two weeks is to set the design matrix equal to the identity matrix. Then your model should give a mean square error which is exactly equal to zero. When you are sure that your codes function well, you can then replace @@ -249,12 +252,17 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)} \end{align*} $$ -
The function will be defined for \( x,y\in [0,1] \). Our first step will +
The function will be defined for \( x,y\in [0,1] \). In a sense, our data are thus scaled to a particular domain for the input values.
+ +Our first step will be to perform an OLS regression analysis of this function, trying out a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, x^2, y^2, xy, \dots] \). We will also include bootstrap first as a resampling technique. After that we will include the cross-validation -technique. As discussed in the lectures for weeks 35 and 36,, we can +technique. +
+ +We can use a uniform distribution to set up the arrays of values for \( x \) and \( y \), or as in the example below just a set of fixed values for \( x \) and \( y \) with a given step size. We will fit a function (for example a @@ -389,26 +397,29 @@ approximately \( 2/3 \) to \( 4/5 \) of the data as training data.
You can easily reuse the solutions to your exercises from week 35 and week 36. See also the lecture slides from week 35 and week 36.
-Write your own code for the Ridge method, either using matrix inversion or the singular value decomposition as done in the previous -exercise. +exercise. The lecture notes from week 35 and 36 contain more information. Furthermore, the numerical exercise from week 36 is something you can reuse here.
Perform the same analysis as you did in the previous exercise but now for different values of \( \lambda \). Compare and -analyze your results with those obtained in parts b-d). Study the +analyze your results with those obtained in part a) with the ordinary least squares method. Study the dependence on \( \lambda \).
+This exercise is essentially a repeat of the previous two ones, but now with Lasso regression. Write either your own code (difficult and optional) or, in this case, -you can also use the functionalities of Scikit-Learn (recommended). +you can also use the functionalities of Scikit-Learn (recommended). Keep in mind that the library Scikit-Learn excludes the intercept by default. Give a critical discussion of the three methods and a judgement of which -model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the bootstrap resampling technique and an analysis of the mean squared error using cross-validation. +model fits the data best.
-v
+This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
@@ -458,9 +469,11 @@ $$We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). . A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
-Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique.
+Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique. +We will only use the simpler ordinary least squares here. +
With a code which does OLS and includes resampling techniques, we will now discuss the bias-variance trade-off in the context of @@ -483,7 +496,7 @@ dataset \( \mathcal{L} \) consisting of the data \( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
-As in part a), we assume that the true data is generated from a noisy model
+As in part d), we assume that the true data is generated from a noisy model
$$ \boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}. @@ -540,12 +553,10 @@ of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method. You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff. -The aim here is to write your own code for another widely popular -resampling technique, the so-called cross-validation method. Again, -before you start with cross-validation approach, you should scale your -data if you think this is needed. +resampling technique, the so-called cross-validation method.
Implement the \( k \)-fold cross-validation algorithm (write your own @@ -559,17 +570,19 @@ you got from your bootstrap code. Comment your results. Try \( 5-10 \) folds. You can also compare your own cross-validation code with the one provided by Scikit-Learn.
+ +In addition to using the ordinary least squares method, you should include both Ridge and Lasso regression.
With our codes functioning and having been tested properly on a simpler function we are now ready to look at real data. We will -essentially repeat in this exercise what was done in exercises 1-5. However, we +essentially repeat in this exercise what was done in exercises a-f. However, we need first to download the data and prepare properly the inputs to our codes. We are going to download digital terrain data from the website https://earthexplorer.usgs.gov/,
-Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2022/Project1/DataFiles
+Or, if you prefer, we have placed selected datafiles at https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2023/Project1/DataFiles
In order to obtain data for a specific region, you need to register as
a user (free) at this website and then decide upon which area you want
diff --git a/doc/Projects/2023/Project1/ipynb/Project1.ipynb b/doc/Projects/2023/Project1/ipynb/Project1.ipynb
index eb44608a1..c867ffa98 100644
--- a/doc/Projects/2023/Project1/ipynb/Project1.ipynb
+++ b/doc/Projects/2023/Project1/ipynb/Project1.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "9b31c43c",
+ "id": "1ceb1efb",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "de414429",
+ "id": "1f73d83b",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
- "id": "24422f72",
+ "id": "df2b7d33",
"metadata": {
"editable": true
},
@@ -44,13 +44,13 @@
"we detail how to write a report. Furthermore, at\n",
" Dh*lnNUV&S=``^P!S4l`mVtl5e4
z8FQ7a$d#WHGtZ3aY@}m|XeES5{t&a6jN$vlU@nG}lryEAt$$n=+QbOODGu!IQ+0pe
zwIub&W>+1C&n9%>f9dA3(@PL3R2CAu;Pf)hSzb=oTyT2HIn$*K*IaOV$qUAHc}9mn
zFFz^B-}7+ULvy&ZLa|nehg}PWm9W&U-7ha=e3zP~*@W-^#*8;*MLF2q)AKB<6ro8?
zSiI@l-JoFm(6`&8SKxxPR7^sfE=q*o(yniBmMf){T}=;Le}w&JUss)d?dsdE?t7Dn
z=)L~+Seea1$BvK=+iuhBtM18dQCI!JKK7W06z>d)NxCVfi*=08%mf~NBVj`*swhit^Do4JFpj}k^Q4K_VmZ*qPe
zi5X9*Ch3d&Ie&fPM#4cn=4MR&{bH#@er8WHg5Usp8ex5RVI?^F8zaUz;=_4iONHlu
zHAceTO)A4bS0uiSgxVHWTVSh0#jfLnl3s`~>wg?X);J|~_X|+QVZJq?Bd}A@FEx2q
zS>Lrj+yzlYQ2767_8rS@(cbRVlp+*VE_h-s#e}KJo^!9n7<1pJEikA93bLJ&9
zxQpZXC(BQ&L$sTw?dYZL3T&aJ%G(*tJ=qz&*d4>ux5``_4Q8^6o
g1m5ce~O15g4x!cn>cjg>CJWF%`De~k2wY*Kq{e6x{Ls5B;HdKn4Ou0
zi{jL9keNS=yE`dOoK
DD;eA3|I9w{}=(9U*JqYiB
zn&V8+URgIlG_!;WgjCHF4N81MhoU)`%!Ap}QH8OBn