The main aim of this project is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, -Ridge regression and finally Lasso regression. -The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. +Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37. + +
+The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
We will first study how to fit polynomials to a specific @@ -218,7 +222,7 @@ beyond the second-order polynomials metioned above and explore which polynomial fits the data best.
-The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
@@ -263,7 +267,7 @@ fig.colorbar(surf, shrink.show() -
We will generate our own dataset for a function @@ -277,7 +281,7 @@ distribution \( \cal{N}(0,1) \). value decomposition from e.g., numpy ) or use your code from homeworks 1 and 2 and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the -confidence intervals of the parameters \( \beta \) by computing their +confidence intervals of the parameters (estimators) \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} @@ -301,9 +305,14 @@ $$ $$
-Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the -function for splitting training data provided by the library Scikit-Learn (make sure you have installed it). -This function is called \( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of Scikit-Learn. +Your code has to include a scaling of the data (for example by +subtracting the mean value, see also homework set 2 for examples) and +a split of the data in training and test data. For this part you can +either write your own code or use for example the function for +splitting training data provided by the library Scikit-Learn (make +sure you have installed it). This function is called +\( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of +Scikit-Learn.
It is normal in essentially all Machine Learning studies to split the @@ -313,7 +322,7 @@ is no explicit recipe for how much data should be included as training data and say test data. An accepted rule of thumb is to use approximately \( 2/3 \) to \( 4/5 \) of the data as training data. -
Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique. @@ -325,6 +334,16 @@ continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks and basically all Machine Learning algorithms. +
+Before you perform an analysis of the bias-variance trade-off on your test data, make +first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and +Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to +indicate possible regions of low/high bias and variance. You will most likely not get an +equally smooth curve! + +
+With this result we move on to the bias-variance trade-off analysis. +
Consider a dataset \( \mathcal{L} \) consisting of the data @@ -355,6 +374,8 @@ $$ C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]. $$ +Here the expected value \( \mathbb{E} \) is the sample value. +
Show that you can rewrite this as $$ @@ -365,25 +386,25 @@ $$ Explain what the terms mean, which one is the bias and which one is the variance and discuss their interpretations. +
+Perform then a bias-variance analysis of the Franke function by +studying the MSE value as function of the complexity of your model. +
Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method.
-However, before you perform an analysis of the bias-variance trade-off on your test data, make -first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and -Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an -equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \). - -
-After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the bootstra resampling technique. +Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
-The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method. -Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier. +The aim here is to write your own code for another widely popular +resampling technique, the so-called cross-validation method. Again, +before you start with cross-validation approach, you should scale your +data and split it in test and training data as you did earlier. Perform a resampling of the data where you split the data in training data and test data using for example @@ -391,7 +412,7 @@ data and test data using for example Implement the \( k \)-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting from the test data. You can compare your own code with that from -Scikit-Learn if needed. You can alternatively write your own bootstrap code. +Scikit-Learn if needed.
Compare the MSE you get from your cross-validation code with the one you got from your bootstrap code. diff --git a/doc/Projects/2020/Project1/html/Project1-bs.html b/doc/Projects/2020/Project1/html/Project1-bs.html index 4f89ee7c5..44db7bb78 100644 --- a/doc/Projects/2020/Project1/html/Project1-bs.html +++ b/doc/Projects/2020/Project1/html/Project1-bs.html @@ -45,12 +45,14 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec0'), - ('Part a): Ordinary Least Square on the Franke function with ' - 'resampling', + ('Part a): Ordinary Least Square (OLS) on the Franke function', 3, None, '___sec1'), - ('Part b): Bias-variance trade-off', 3, None, '___sec2'), + ('Part b): Bias-variance trade-off and resamplng techniques', + 3, + None, + '___sec2'), ('Part c) Cross-validation as resampling techniques, adding more ' 'complexity', 3, @@ -119,8 +121,8 @@ MathJax.Hub.Config({ Contents
The main aim of this project is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, -Ridge regression and finally Lasso regression. -The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. +Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37. + +
+The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
We will first study how to fit polynomials to a specific @@ -218,7 +222,7 @@ beyond the second-order polynomials metioned above and explore which polynomial fits the data best.
-The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
@@ -263,7 +267,7 @@ fig.colorbar(surf, shrink.show() -
We will generate our own dataset for a function @@ -277,7 +281,7 @@ distribution \( \cal{N}(0,1) \). value decomposition from e.g., numpy ) or use your code from homeworks 1 and 2 and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the -confidence intervals of the parameters \( \beta \) by computing their +confidence intervals of the parameters (estimators) \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} @@ -301,9 +305,14 @@ $$ $$
-Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the -function for splitting training data provided by the library Scikit-Learn (make sure you have installed it). -This function is called \( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of Scikit-Learn. +Your code has to include a scaling of the data (for example by +subtracting the mean value, see also homework set 2 for examples) and +a split of the data in training and test data. For this part you can +either write your own code or use for example the function for +splitting training data provided by the library Scikit-Learn (make +sure you have installed it). This function is called +\( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of +Scikit-Learn.
It is normal in essentially all Machine Learning studies to split the @@ -313,7 +322,7 @@ is no explicit recipe for how much data should be included as training data and say test data. An accepted rule of thumb is to use approximately \( 2/3 \) to \( 4/5 \) of the data as training data. -
Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique. @@ -325,6 +334,16 @@ continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks and basically all Machine Learning algorithms. +
+Before you perform an analysis of the bias-variance trade-off on your test data, make +first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and +Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to +indicate possible regions of low/high bias and variance. You will most likely not get an +equally smooth curve! + +
+With this result we move on to the bias-variance trade-off analysis. +
Consider a dataset \( \mathcal{L} \) consisting of the data @@ -355,6 +374,8 @@ $$ C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]. $$ +Here the expected value \( \mathbb{E} \) is the sample value. +
Show that you can rewrite this as $$ @@ -365,25 +386,25 @@ $$ Explain what the terms mean, which one is the bias and which one is the variance and discuss their interpretations. +
+Perform then a bias-variance analysis of the Franke function by +studying the MSE value as function of the complexity of your model. +
Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method.
-However, before you perform an analysis of the bias-variance trade-off on your test data, make -first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and -Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an -equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \). - -
-After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the bootstra resampling technique. +Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
-The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method. -Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier. +The aim here is to write your own code for another widely popular +resampling technique, the so-called cross-validation method. Again, +before you start with cross-validation approach, you should scale your +data and split it in test and training data as you did earlier. Perform a resampling of the data where you split the data in training data and test data using for example @@ -391,7 +412,7 @@ data and test data using for example Implement the \( k \)-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting from the test data. You can compare your own code with that from -Scikit-Learn if needed. You can alternatively write your own bootstrap code. +Scikit-Learn if needed.
Compare the MSE you get from your cross-validation code with the one you got from your bootstrap code. diff --git a/doc/Projects/2020/Project1/html/Project1.html b/doc/Projects/2020/Project1/html/Project1.html index 90e66f282..432642e56 100644 --- a/doc/Projects/2020/Project1/html/Project1.html +++ b/doc/Projects/2020/Project1/html/Project1.html @@ -44,12 +44,14 @@ div { text-align: justify; text-justify: inter-word; } 2, None, '___sec0'), - ('Part a): Ordinary Least Square on the Franke function with ' - 'resampling', + ('Part a): Ordinary Least Square (OLS) on the Franke function', 3, None, '___sec1'), - ('Part b): Bias-variance trade-off', 3, None, '___sec2'), + ('Part b): Bias-variance trade-off and resamplng techniques', + 3, + None, + '___sec2'), ('Part c) Cross-validation as resampling techniques, adding more ' 'complexity', 3, @@ -128,8 +130,10 @@ MathJax.Hub.Config({
The main aim of this project is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, -Ridge regression and finally Lasso regression. -The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. +Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37. + +
+The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37.
We will first study how to fit polynomials to a specific @@ -173,7 +177,7 @@ beyond the second-order polynomials metioned above and explore which polynomial fits the data best.
-The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
@@ -218,7 +222,7 @@ fig.colorbar(surf, shrink.show() -
We will generate our own dataset for a function @@ -232,7 +236,7 @@ distribution \( \cal{N}(0,1) \). value decomposition from e.g., numpy ) or use your code from homeworks 1 and 2 and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the -confidence intervals of the parameters \( \beta \) by computing their +confidence intervals of the parameters (estimators) \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} @@ -256,9 +260,14 @@ $$ $$
-Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the -function for splitting training data provided by the library Scikit-Learn (make sure you have installed it). -This function is called \( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of Scikit-Learn. +Your code has to include a scaling of the data (for example by +subtracting the mean value, see also homework set 2 for examples) and +a split of the data in training and test data. For this part you can +either write your own code or use for example the function for +splitting training data provided by the library Scikit-Learn (make +sure you have installed it). This function is called +\( train\_test\_split \). Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of +Scikit-Learn.
It is normal in essentially all Machine Learning studies to split the @@ -268,7 +277,7 @@ is no explicit recipe for how much data should be included as training data and say test data. An accepted rule of thumb is to use approximately \( 2/3 \) to \( 4/5 \) of the data as training data. -
Our aim here is to study the bias-variance trade-off by implementing the bootstrap resampling technique. @@ -280,6 +289,16 @@ continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks and basically all Machine Learning algorithms. +
+Before you perform an analysis of the bias-variance trade-off on your test data, make +first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and +Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to +indicate possible regions of low/high bias and variance. You will most likely not get an +equally smooth curve! + +
+With this result we move on to the bias-variance trade-off analysis. +
Consider a dataset \( \mathcal{L} \) consisting of the data @@ -310,6 +329,8 @@ $$ C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]. $$ +Here the expected value \( \mathbb{E} \) is the sample value. +
Show that you can rewrite this as $$ @@ -320,25 +341,25 @@ $$ Explain what the terms mean, which one is the bias and which one is the variance and discuss their interpretations. +
+Perform then a bias-variance analysis of the Franke function by +studying the MSE value as function of the complexity of your model. +
Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method.
-However, before you perform an analysis of the bias-variance trade-off on your test data, make -first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and -Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an -equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \). - -
-After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the bootstra resampling technique. +Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
-The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method. -Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier. +The aim here is to write your own code for another widely popular +resampling technique, the so-called cross-validation method. Again, +before you start with cross-validation approach, you should scale your +data and split it in test and training data as you did earlier. Perform a resampling of the data where you split the data in training data and test data using for example @@ -346,7 +367,7 @@ data and test data using for example Implement the \( k \)-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting from the test data. You can compare your own code with that from -Scikit-Learn if needed. You can alternatively write your own bootstrap code. +Scikit-Learn if needed.
Compare the MSE you get from your cross-validation code with the one you got from your bootstrap code. diff --git a/doc/Projects/2020/Project1/ipynb/ipynb-Project1-src.tar.gz b/doc/Projects/2020/Project1/ipynb/ipynb-Project1-src.tar.gz index 2f9e4fabb..b2b4f085b 100644 Binary files a/doc/Projects/2020/Project1/ipynb/ipynb-Project1-src.tar.gz and b/doc/Projects/2020/Project1/ipynb/ipynb-Project1-src.tar.gz differ diff --git a/doc/Projects/2020/Project1/pdf/Project1.p.tex b/doc/Projects/2020/Project1/pdf/Project1.p.tex index 67151ae52..cc7e020b4 100644 --- a/doc/Projects/2020/Project1/pdf/Project1.p.tex +++ b/doc/Projects/2020/Project1/pdf/Project1.p.tex @@ -166,8 +166,9 @@ Sep 1, 2020 The main aim of this project is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, -Ridge regression and finally Lasso regression. -The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. +Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37. + +The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37. We will first study how to fit polynomials to a specific two-dimensional function called \href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's @@ -206,7 +207,7 @@ beyond the second-order polynomials metioned above and explore which polynomial fits the data best. -The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +The Python code for the Franke function is included here (it performs also a three-dimensional plot of it) \bpycod from mpl_toolkits.mplot3d import Axes3D import matplotlib.pyplot as plt @@ -251,7 +252,7 @@ plt.show() \epycod -\paragraph{Part a): Ordinary Least Square on the Franke function with resampling.} +\paragraph{Part a): Ordinary Least Square (OLS) on the Franke function.} We will generate our own dataset for a function $\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function $f(x,y)$ is the Franke function. You should explore also the addition @@ -262,7 +263,7 @@ distribution $\cal{N}(0,1)$. value decomposition from e.g., \textbf{numpy} ) or use your code from homeworks 1 and 2 and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the -confidence intervals of the parameters $\beta$ by computing their +\href{{https://en.wikipedia.org/wiki/Confidence_interval}}{confidence intervals} of the parameters (estimators) $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) \[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} @@ -283,9 +284,14 @@ where we have defined the mean value of $\hat{y}$ as \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. \] -Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the -function for splitting training data provided by the library \textbf{Scikit-Learn} (make sure you have installed it). -This function is called $train\_test\_split$. Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of \textbf{Scikit-Learn}. +Your code has to include a scaling of the data (for example by +subtracting the mean value, see also \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2} for examples) and +a split of the data in training and test data. For this part you can +either write your own code or use for example the function for +splitting training data provided by the library \textbf{Scikit-Learn} (make +sure you have installed it). This function is called +$train\_test\_split$. Similarly, and see the solution to \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2, exercise 2}, you can use the data normalization functionality of +\textbf{Scikit-Learn}. It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (eventually also an additional @@ -299,7 +305,7 @@ approximately $2/3$ to $4/5$ of the data as training data. -\paragraph{Part b): Bias-variance trade-off.} +\paragraph{Part b): Bias-variance trade-off and resamplng techniques.} Our aim here is to study the bias-variance trade-off by implementing the \textbf{bootstrap} resampling technique. With a code which does OLS and includes resampling techniques, @@ -308,6 +314,14 @@ continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks and basically all Machine Learning algorithms. +Before you perform an analysis of the bias-variance trade-off on your test data, make +first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and +Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to +indicate possible regions of low/high bias and variance. You will most likely not get an +equally smooth curve! + +With this result we move on to the bias-variance trade-off analysis. + Consider a dataset $\mathcal{L}$ consisting of the data $\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. @@ -332,6 +346,7 @@ squared error via the so-called cost function \[ C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. \] +Here the expected value $\mathbb{E}$ is the sample value. Show that you can rewrite this as \[ @@ -341,21 +356,21 @@ Show that you can rewrite this as Explain what the terms mean, which one is the bias and which one is the variance and discuss their interpretations. +Perform then a bias-variance analysis of the Franke function by +studying the MSE value as function of the complexity of your model. Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method. -However, before you perform an analysis of the bias-variance trade-off on your test data, make -first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and -Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an -equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$. +Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$. -After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the \textbf{bootstra} resampling technique. \paragraph{Part c) Cross-validation as resampling techniques, adding more complexity.} -The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method. -Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier. +The aim here is to write your own code for another widely popular +resampling technique, the so-called cross-validation method. Again, +before you start with cross-validation approach, you should scale your +data and split it in test and training data as you did earlier. Perform a resampling of the data where you split the data in training data and test data using for example @@ -364,7 +379,7 @@ data and test data using for example Implement the $k$-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting from the test data. You can compare your own code with that from -\textbf{Scikit-Learn} if needed. You can alternatively write your own bootstrap code. +\textbf{Scikit-Learn} if needed. Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code. You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}. diff --git a/doc/Projects/2020/Project1/pdf/Project1.pdf b/doc/Projects/2020/Project1/pdf/Project1.pdf index 4ce5df235..a001b89ba 100644 Binary files a/doc/Projects/2020/Project1/pdf/Project1.pdf and b/doc/Projects/2020/Project1/pdf/Project1.pdf differ diff --git a/doc/Projects/2020/Project1/pdf/Project1.tex b/doc/Projects/2020/Project1/pdf/Project1.tex index 43a7ec7c1..db8035a2e 100644 --- a/doc/Projects/2020/Project1/pdf/Project1.tex +++ b/doc/Projects/2020/Project1/pdf/Project1.tex @@ -136,8 +136,9 @@ Sep 1, 2020 The main aim of this project is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, -Ridge regression and finally Lasso regression. -The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. +Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37. + +The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37. We will first study how to fit polynomials to a specific two-dimensional function called \href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's @@ -176,7 +177,7 @@ beyond the second-order polynomials metioned above and explore which polynomial fits the data best. -The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +The Python code for the Franke function is included here (it performs also a three-dimensional plot of it) \begin{verbatim} from mpl_toolkits.mplot3d import Axes3D import matplotlib.pyplot as plt @@ -221,7 +222,7 @@ plt.show() \end{verbatim} -\paragraph{Part a): Ordinary Least Square on the Franke function with resampling.} +\paragraph{Part a): Ordinary Least Square (OLS) on the Franke function.} We will generate our own dataset for a function $\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function $f(x,y)$ is the Franke function. You should explore also the addition @@ -232,7 +233,7 @@ distribution $\cal{N}(0,1)$. value decomposition from e.g., \textbf{numpy} ) or use your code from homeworks 1 and 2 and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the -confidence intervals of the parameters $\beta$ by computing their +\href{{https://en.wikipedia.org/wiki/Confidence_interval}}{confidence intervals} of the parameters (estimators) $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) \[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} @@ -253,9 +254,14 @@ where we have defined the mean value of $\hat{y}$ as \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. \] -Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the -function for splitting training data provided by the library \textbf{Scikit-Learn} (make sure you have installed it). -This function is called $train\_test\_split$. Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of \textbf{Scikit-Learn}. +Your code has to include a scaling of the data (for example by +subtracting the mean value, see also \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2} for examples) and +a split of the data in training and test data. For this part you can +either write your own code or use for example the function for +splitting training data provided by the library \textbf{Scikit-Learn} (make +sure you have installed it). This function is called +$train\_test\_split$. Similarly, and see the solution to \href{{https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html}}{homework set 2, exercise 2}, you can use the data normalization functionality of +\textbf{Scikit-Learn}. It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (eventually also an additional @@ -269,7 +275,7 @@ approximately $2/3$ to $4/5$ of the data as training data. -\paragraph{Part b): Bias-variance trade-off.} +\paragraph{Part b): Bias-variance trade-off and resamplng techniques.} Our aim here is to study the bias-variance trade-off by implementing the \textbf{bootstrap} resampling technique. With a code which does OLS and includes resampling techniques, @@ -278,6 +284,14 @@ continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks and basically all Machine Learning algorithms. +Before you perform an analysis of the bias-variance trade-off on your test data, make +first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and +Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to +indicate possible regions of low/high bias and variance. You will most likely not get an +equally smooth curve! + +With this result we move on to the bias-variance trade-off analysis. + Consider a dataset $\mathcal{L}$ consisting of the data $\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. @@ -302,6 +316,7 @@ squared error via the so-called cost function \[ C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. \] +Here the expected value $\mathbb{E}$ is the sample value. Show that you can rewrite this as \[ @@ -311,21 +326,21 @@ Show that you can rewrite this as Explain what the terms mean, which one is the bias and which one is the variance and discuss their interpretations. +Perform then a bias-variance analysis of the Franke function by +studying the MSE value as function of the complexity of your model. Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method. -However, before you perform an analysis of the bias-variance trade-off on your test data, make -first a figure similar to Fig.~2.11 of Hastie, Tibshirani, and -Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an -equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$. +Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$. -After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the \textbf{bootstra} resampling technique. \paragraph{Part c) Cross-validation as resampling techniques, adding more complexity.} -The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method. -Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier. +The aim here is to write your own code for another widely popular +resampling technique, the so-called cross-validation method. Again, +before you start with cross-validation approach, you should scale your +data and split it in test and training data as you did earlier. Perform a resampling of the data where you split the data in training data and test data using for example @@ -334,7 +349,7 @@ data and test data using for example Implement the $k$-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting from the test data. You can compare your own code with that from -\textbf{Scikit-Learn} if needed. You can alternatively write your own bootstrap code. +\textbf{Scikit-Learn} if needed. Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code. You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}. @@ -408,7 +423,7 @@ Austfjell, again in Norway. Feel free to produce your own terrain data. -Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your research area or simply a data set you found interesting. See for example \href{{https://www.kaggle.com/datasets}}{kaggle.com} for examples. +Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example \href{{https://www.kaggle.com/datasets}}{kaggle.com} for examples. \paragraph{Part g) OLS, Ridge and Lasso regression with resampling.} Our final part deals with the parameterization of your digital terrain diff --git a/doc/src/Projects/2020/Project1/Project1.do.txt b/doc/src/Projects/2020/Project1/Project1.do.txt index 6fd9ddcb6..53dc0b901 100644 --- a/doc/src/Projects/2020/Project1/Project1.do.txt +++ b/doc/src/Projects/2020/Project1/Project1.do.txt @@ -8,8 +8,9 @@ DATE: today The main aim of this project is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, -Ridge regression and finally Lasso regression. -The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. +Ridge regression and finally Lasso regression. Ridge regression will be discussed during the Friday lecture of week 36 while Lasso Regression will be discussed during the lectures of week 37. + +The methods are in turn combined with resampling techniques like the bootstrap method and cross validation. These are discussed during weeks 36 and 37. We will first study how to fit polynomials to a specific two-dimensional function called "Franke's @@ -50,7 +51,7 @@ beyond the second-order polynomials metioned above and explore which polynomial fits the data best. -The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +The Python code for the Franke function is included here (it performs also a three-dimensional plot of it) !bc pycod from mpl_toolkits.mplot3d import Axes3D import matplotlib.pyplot as plt @@ -95,7 +96,7 @@ plt.show() !ec -=== Part a): Ordinary Least Square on the Franke function with resampling === +=== Part a): Ordinary Least Square (OLS) on the Franke function === We will generate our own dataset for a function $\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function @@ -107,7 +108,7 @@ distribution $\cal{N}(0,1)$. value decomposition from e.g., _numpy_ ) or use your code from homeworks 1 and 2 and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the -confidence intervals of the parameters $\beta$ by computing their +"confidence intervals":"https://en.wikipedia.org/wiki/Confidence_interval" of the parameters (estimators) $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) !bt @@ -134,9 +135,14 @@ where we have defined the mean value of $\hat{y}$ as \] !et -Your code has to include a scaling of the data (for example by subtracting the mean value, see also homework set 2 for examples) and a split of the data in training and test data. For this part you can either write your own code or use for example the -function for splitting training data provided by the library _Scikit-Learn_ (make sure you have installed it). -This function is called $train\_test\_split$. Similarly, and see the solution to homework set 2, exercise 2, you can use the data normalization functionality of _Scikit-Learn_. +Your code has to include a scaling of the data (for example by +subtracting the mean value, see also "homework set 2":"https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html" for examples) and +a split of the data in training and test data. For this part you can +either write your own code or use for example the function for +splitting training data provided by the library _Scikit-Learn_ (make +sure you have installed it). This function is called +$train\_test\_split$. Similarly, and see the solution to "homework set 2, exercise 2":"https://compphysics.github.io/MachineLearning/doc/Projects/2020/hw2/html/hw2-bs.html", you can use the data normalization functionality of +_Scikit-Learn_. It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (eventually also an additional @@ -150,7 +156,7 @@ approximately $2/3$ to $4/5$ of the data as training data. -=== Part b): Bias-variance trade-off === +=== Part b): Bias-variance trade-off and resamplng techniques === Our aim here is to study the bias-variance trade-off by implementing the _bootstrap_ resampling technique. @@ -160,6 +166,14 @@ continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks and basically all Machine Learning algorithms. +Before you perform an analysis of the bias-variance trade-off on your test data, make +first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and +Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to +indicate possible regions of low/high bias and variance. You will most likely not get an +equally smooth curve! + +With this result we move on to the bias-variance trade-off analysis. + Consider a dataset $\mathcal{L}$ consisting of the data $\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. @@ -188,6 +202,7 @@ squared error via the so-called cost function C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. \] !et +Here the expected value $\mathbb{E}$ is the sample value. Show that you can rewrite this as !bt @@ -199,23 +214,23 @@ Show that you can rewrite this as Explain what the terms mean, which one is the bias and which one is the variance and discuss their interpretations. +Perform then a bias-variance analysis of the Franke function by +studying the MSE value as function of the complexity of your model. Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the _bootstrap_ resampling method. -However, before you perform an analysis of the bias-variance trade-off on your test data, make -first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and -Friedman. Figure 2.11 of this reference displays only the test and training MSEs (and bias-variance analysis of the test data) while indicating possible regions of low/high bias and variance. You will most likely not get an -equally smooth curve! Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$. +Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$. -After having produced a curve similar to Figure 2.11, perform then a bias-variance analysis of your test data. Here you should use the _bootstra_ resampling technique. === Part c) Cross-validation as resampling techniques, adding more complexity === -The aim here is to write our own code for another widely popular resampling technique, the so-called cross-validation method. -Again, before you start with cross-validation approach, you should scale your data and split it in test and training data as you did earlier. +The aim here is to write your own code for another widely popular +resampling technique, the so-called cross-validation method. Again, +before you start with cross-validation approach, you should scale your +data and split it in test and training data as you did earlier. Perform a resampling of the data where you split the data in training data and test data using for example @@ -224,7 +239,7 @@ data and test data using for example Implement the $k$-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting from the test data. You can compare your own code with that from -_Scikit-Learn_ if needed. You can alternatively write your own bootstrap code. +_Scikit-Learn_ if needed. Compare the MSE you get from your cross-validation code with the one you got from your _bootstrap_ code. You can also compare your own cross-validation code with the one provided by _Scikit-Learn_. diff --git a/doc/src/Projects/2020/Project1/make.sh b/doc/src/Projects/2020/Project1/make.sh index 740fedbc3..c050a5717 100755 --- a/doc/src/Projects/2020/Project1/make.sh +++ b/doc/src/Projects/2020/Project1/make.sh @@ -38,7 +38,7 @@ system doconce split_html $html.html --method=split --pagination --nav_button=bo # Ordinary plain LaTeX document system doconce format pdflatex $name --print_latex_style=trac --latex_admon=paragraph $opt -system doconce ptex2tex $name envir=print +system doconce ptex2tex $name envir=verbatim # Add special packages doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex doconce replace 'section{' 'section*{' $name.tex