updated book
This commit is contained in:
Morten Hjorth-Jensen
@@ -1014,3 +1014,326 @@ plt.show()
|
||||
!ec
|
||||
|
||||
|
||||
===== Exercises and Projects =====
|
||||
|
||||
|
||||
|
||||
The main aim of this project is to study in more detail various
|
||||
regression methods, including the Ordinary Least Squares (OLS) method,
|
||||
The total score is _100_ points. Each subtask has its own final score.
|
||||
|
||||
|
||||
We will first study how to fit polynomials to a specific
|
||||
two-dimensional function called "Franke's
|
||||
function":"http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf". This
|
||||
is a function which has been widely used when testing various
|
||||
interpolation and fitting algorithms. Furthermore, after having
|
||||
established the model and the method, we will employ resamling
|
||||
techniques such as cross-validation and/or bootstrap in order to perform a
|
||||
proper assessment of our models. We will also study in detail the
|
||||
so-called Bias-Variance trade off.
|
||||
|
||||
|
||||
The Franke function, which is a weighted sum of four exponentials reads as follows
|
||||
!bt
|
||||
\begin{align*}
|
||||
f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\
|
||||
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
|
||||
\end{align*}
|
||||
!et
|
||||
|
||||
The function will be defined for $x,y\in [0,1]$. Our first step will
|
||||
be to perform an OLS regression analysis of this function, trying out
|
||||
a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,
|
||||
x^2, y^2, xy, \dots]$. We will also include bootstrap first as
|
||||
a resampling technique. After that we will include the cross-validation technique. As in homeworks 1 and 2, we can use a uniform
|
||||
distribution to set up the arrays of values for $x$ and $y$, or as in
|
||||
the example below just a set of fixed
|
||||
values for $x$ and $y$ with a given step
|
||||
size. We will fit a
|
||||
function (for example a polynomial) of $x$ and $y$. Thereafter we
|
||||
will repeat much of the same procedure using the Ridge and Lasso
|
||||
regression methods, introducing thus a dependence on the bias
|
||||
(penalty) $\lambda$.
|
||||
|
||||
Finally we are going to use (real) digital terrain data and try to
|
||||
reproduce these data using the same methods. We will also try to go
|
||||
beyond the second-order polynomials metioned above and explore
|
||||
which polynomial fits the data best.
|
||||
|
||||
|
||||
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
|
||||
!bc pycod
|
||||
from mpl_toolkits.mplot3d import Axes3D
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib import cm
|
||||
from matplotlib.ticker import LinearLocator, FormatStrFormatter
|
||||
import numpy as np
|
||||
from random import random, seed
|
||||
|
||||
fig = plt.figure()
|
||||
ax = fig.gca(projection='3d')
|
||||
|
||||
# Make data.
|
||||
x = np.arange(0, 1, 0.05)
|
||||
y = np.arange(0, 1, 0.05)
|
||||
x, y = np.meshgrid(x,y)
|
||||
|
||||
|
||||
def FrankeFunction(x,y):
|
||||
term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
|
||||
term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
|
||||
term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
|
||||
term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
|
||||
return term1 + term2 + term3 + term4
|
||||
|
||||
|
||||
z = FrankeFunction(x, y)
|
||||
|
||||
# Plot the surface.
|
||||
surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,
|
||||
linewidth=0, antialiased=False)
|
||||
|
||||
# Customize the z axis.
|
||||
ax.set_zlim(-0.10, 1.40)
|
||||
ax.zaxis.set_major_locator(LinearLocator(10))
|
||||
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
|
||||
|
||||
# Add a color bar which maps values to colors.
|
||||
fig.colorbar(surf, shrink=0.5, aspect=5)
|
||||
|
||||
plt.show()
|
||||
|
||||
!ec
|
||||
|
||||
|
||||
=== Exercise: Ordinary Least Square (OLS) on the Franke function ===
|
||||
|
||||
We will generate our own dataset for a function
|
||||
$\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function
|
||||
$f(x,y)$ is the Franke function. You should explore also the addition
|
||||
of an added stochastic noise to this function using the normal
|
||||
distribution $N(0,1)$.
|
||||
|
||||
*Write your own code* (using either a matrix inversion or a singular
|
||||
value decomposition from e.g., _numpy_ ) or use your code from
|
||||
homeworks 1 and 2 and perform a standard least square regression
|
||||
analysis using polynomials in $x$ and $y$ up to fifth order. Find the
|
||||
"confidence intervals":"https://en.wikipedia.org/wiki/Confidence_interval" of the parameters (estimators) $\beta$ by computing their
|
||||
variances, evaluate the Mean Squared error (MSE)
|
||||
|
||||
!bt
|
||||
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
\]
|
||||
!et
|
||||
|
||||
and the $R^2$ score function. If $\tilde{\hat{y}}_i$ is the predicted
|
||||
value of the $i-th$ sample and $y_i$ is the corresponding true value,
|
||||
then the score $R^2$ is defined as
|
||||
|
||||
!bt
|
||||
\[
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
\]
|
||||
!et
|
||||
|
||||
where we have defined the mean value of $\hat{y}$ as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
\]
|
||||
!et
|
||||
|
||||
Your code has to include a scaling of the data (for example by
|
||||
subtracting the mean value), and
|
||||
a split of the data in training and test data. For this exercise you can
|
||||
either write your own code or use for example the function for
|
||||
splitting training data provided by the library _Scikit-Learn_ (make
|
||||
sure you have installed it). This function is called
|
||||
$train\_test\_split$. _You should present a critical discussion of why and how you have scaled or not scaled the data_.
|
||||
|
||||
It is normal in essentially all Machine Learning studies to split the
|
||||
data in a training set and a test set (eventually also an additional
|
||||
validation set). There
|
||||
is no explicit recipe for how much data should be included as training
|
||||
data and say test data. An accepted rule of thumb is to use
|
||||
approximately $2/3$ to $4/5$ of the data as training data.
|
||||
|
||||
|
||||
You can easily reuse the solutions to your exercises from week 35 and week 36.
|
||||
|
||||
|
||||
|
||||
=== Exercise: Bias-variance trade-off and resampling techniques ===
|
||||
|
||||
Our aim here is to study the bias-variance trade-off by implementing the _bootstrap_ resampling technique.
|
||||
|
||||
With a code which does OLS and includes resampling techniques,
|
||||
we will now discuss the bias-variance trade-off in the context of
|
||||
continuous predictions such as regression. However, many of the
|
||||
intuitions and ideas discussed here also carry over to classification
|
||||
tasks and basically all Machine Learning algorithms.
|
||||
|
||||
Before you perform an analysis of the bias-variance trade-off on your test data, make
|
||||
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
|
||||
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
|
||||
indicate possible regions of low/high bias and variance. You will most likely not get an
|
||||
equally smooth curve!
|
||||
|
||||
With this result we move on to the bias-variance trade-off analysis.
|
||||
|
||||
Consider a
|
||||
dataset $\mathcal{L}$ consisting of the data
|
||||
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
|
||||
|
||||
Let us assume that the true data is generated from a noisy model
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}.
|
||||
\]
|
||||
!et
|
||||
|
||||
Here $\epsilon$ is normally distributed with mean zero and standard
|
||||
deviation $\sigma^2$.
|
||||
|
||||
In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function $f$ in terms of the parameters
|
||||
$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model,
|
||||
that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$.
|
||||
|
||||
The parameters $\bm{\beta}$ are in turn found by optimizing the means
|
||||
squared error via the so-called cost function
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
|
||||
\]
|
||||
!et
|
||||
Here the expected value $\mathbb{E}$ is the sample value.
|
||||
|
||||
Show that you can rewrite this as
|
||||
!bt
|
||||
\[
|
||||
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2.
|
||||
\]
|
||||
!et
|
||||
|
||||
Explain what the terms mean, which one is the bias and which one is
|
||||
the variance and discuss their interpretations.
|
||||
|
||||
Perform then a bias-variance analysis of the Franke function by
|
||||
studying the MSE value as function of the complexity of your model.
|
||||
|
||||
Discuss the bias and variance trade-off as function
|
||||
of your model complexity (the degree of the polynomial) and the number
|
||||
of data points, and possibly also your training and test data using the _bootstrap_ resampling method.
|
||||
|
||||
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
|
||||
|
||||
|
||||
=== Exercise: Cross-validation as resampling techniques, adding more complexity ===
|
||||
|
||||
|
||||
The aim here is to write your own code for another widely popular
|
||||
resampling technique, the so-called cross-validation method. Again,
|
||||
before you start with cross-validation approach, you should scale your
|
||||
data.
|
||||
|
||||
Implement the $k$-fold cross-validation algorithm (write your own
|
||||
code) and evaluate again the MSE function resulting
|
||||
from the test folds. You can compare your own code with that from
|
||||
_Scikit-Learn_ if needed.
|
||||
|
||||
Compare the MSE you get from your cross-validation code with the one
|
||||
you got from your _bootstrap_ code. Comment your results. Try $5-10$
|
||||
folds. You can also compare your own cross-validation code with the
|
||||
one provided by _Scikit-Learn_.
|
||||
|
||||
|
||||
=== Exercise: Ridge Regression on the Franke function with resampling ===
|
||||
|
||||
Write your own code for the Ridge method, either using matrix
|
||||
inversion or the singular value decomposition as done in the previous
|
||||
exercise. Perform the same bootstrap analysis as in the
|
||||
Exercise 2 (for the same polynomials) and the cross-validation in exercise 3 but now for different values of $\lambda$. Compare and
|
||||
analyze your results with those obtained in exercises 1-3. Study the
|
||||
dependence on $\lambda$.
|
||||
|
||||
Study also the bias-variance trade-off as function of various values of
|
||||
the parameter $\lambda$. For the bias-variance trade-off, use the _bootstrap_ resampling method. Comment your results.
|
||||
|
||||
=== Exercise: Lasso Regression on the Franke function with resampling ===
|
||||
|
||||
This exercise is essentially a repeat of the previous two ones, but now
|
||||
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
|
||||
you can also use the functionalities of _Scikit-Learn_ (recommended).
|
||||
Give a
|
||||
critical discussion of the three methods and a judgement of which
|
||||
model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the _bootstrap_ resampling technique and an analysis of the mean squared error using cross-validation.
|
||||
|
||||
=== Exercise: Analysis of real data ===
|
||||
|
||||
With our codes functioning and having been tested properly on a
|
||||
simpler function we are now ready to look at real data. We will
|
||||
essentially repeat in this exercise what was done in exercises 1-5. However, we
|
||||
need first to download the data and prepare properly the inputs to our
|
||||
codes. We are going to download digital terrain data from the website
|
||||
URL:"https://earthexplorer.usgs.gov/",
|
||||
|
||||
Or, if you prefer, we have placed selected datafiles at URL:"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2021/Project1/DataFiles"
|
||||
|
||||
In order to obtain data for a specific region, you need to register as
|
||||
a user (free) at this website and then decide upon which area you want
|
||||
to fetch the digital terrain data from. In order to be able to read
|
||||
the data properly, you need to specify that the format should be _SRTM
|
||||
Arc-Second Global_ and download the data as a _GeoTIF_ file. The
|
||||
files are then stored in *tif* format which can be imported into a
|
||||
Python program using
|
||||
|
||||
!bc pycod
|
||||
scipy.misc.imread
|
||||
!ec
|
||||
|
||||
Here is a simple part of a Python code which reads and plots the data
|
||||
from such files
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
from imageio import imread
|
||||
import matplotlib.pyplot as plt
|
||||
from mpl_toolkits.mplot3d import Axes3D
|
||||
from matplotlib import cm
|
||||
|
||||
# Load the terrain
|
||||
terrain1 = imread('SRTM_data_Norway_1.tif')
|
||||
# Show the terrain
|
||||
plt.figure()
|
||||
plt.title('Terrain over Norway 1')
|
||||
plt.imshow(terrain1, cmap='gray')
|
||||
plt.xlabel('X')
|
||||
plt.ylabel('Y')
|
||||
plt.show()
|
||||
!ec
|
||||
|
||||
If you should have problems in downloading the digital terrain data,
|
||||
we provide two examples under the data folder of project 1. One is
|
||||
from a region close to Stavanger in Norway and the other Møsvatn
|
||||
Austfjell, again in Norway.
|
||||
Feel free to produce your own terrain data.
|
||||
|
||||
|
||||
Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example "kaggle.com":"https://www.kaggle.com/datasets" for examples.
|
||||
|
||||
|
||||
Our final part deals with the parameterization of your digital terrain
|
||||
data (or your own data). We will apply all three methods for linear regression, the same type (or higher order) of polynomial
|
||||
approximation and cross-validation as resampling technique to evaluate which
|
||||
model fits the data best.
|
||||
|
||||
At the end, you should present a critical evaluation of your results
|
||||
and discuss the applicability of these regression methods to the type
|
||||
of data presented here (either the terrain data we propose or other data sets).
|
||||
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
@@ -1282,6 +1282,433 @@
|
||||
"plt.legend()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercises and Projects\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The main aim of this project is to study in more detail various\n",
|
||||
"regression methods, including the Ordinary Least Squares (OLS) method,\n",
|
||||
"The total score is **100** points. Each subtask has its own final score.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"We will first study how to fit polynomials to a specific\n",
|
||||
"two-dimensional function called [Franke's\n",
|
||||
"function](http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf). This\n",
|
||||
"is a function which has been widely used when testing various\n",
|
||||
"interpolation and fitting algorithms. Furthermore, after having\n",
|
||||
"established the model and the method, we will employ resamling\n",
|
||||
"techniques such as cross-validation and/or bootstrap in order to perform a\n",
|
||||
"proper assessment of our models. We will also study in detail the\n",
|
||||
"so-called Bias-Variance trade off.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The Franke function, which is a weighted sum of four exponentials reads as follows"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{align*}\n",
|
||||
"f(x,y) &= \\frac{3}{4}\\exp{\\left(-\\frac{(9x-2)^2}{4} - \\frac{(9y-2)^2}{4}\\right)}+\\frac{3}{4}\\exp{\\left(-\\frac{(9x+1)^2}{49}- \\frac{(9y+1)}{10}\\right)} \\\\\n",
|
||||
"&+\\frac{1}{2}\\exp{\\left(-\\frac{(9x-7)^2}{4} - \\frac{(9y-3)^2}{4}\\right)} -\\frac{1}{5}\\exp{\\left(-(9x-4)^2 - (9y-7)^2\\right) }.\n",
|
||||
"\\end{align*}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The function will be defined for $x,y\\in [0,1]$. Our first step will\n",
|
||||
"be to perform an OLS regression analysis of this function, trying out\n",
|
||||
"a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,\n",
|
||||
"x^2, y^2, xy, \\dots]$. We will also include bootstrap first as\n",
|
||||
"a resampling technique. After that we will include the cross-validation technique. As in homeworks 1 and 2, we can use a uniform\n",
|
||||
"distribution to set up the arrays of values for $x$ and $y$, or as in\n",
|
||||
"the example below just a set of fixed \n",
|
||||
"values for $x$ and $y$ with a given step\n",
|
||||
"size. We will fit a\n",
|
||||
"function (for example a polynomial) of $x$ and $y$. Thereafter we\n",
|
||||
"will repeat much of the same procedure using the Ridge and Lasso\n",
|
||||
"regression methods, introducing thus a dependence on the bias\n",
|
||||
"(penalty) $\\lambda$.\n",
|
||||
"\n",
|
||||
"Finally we are going to use (real) digital terrain data and try to\n",
|
||||
"reproduce these data using the same methods. We will also try to go\n",
|
||||
"beyond the second-order polynomials metioned above and explore \n",
|
||||
"which polynomial fits the data best.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from mpl_toolkits.mplot3d import Axes3D\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from matplotlib import cm\n",
|
||||
"from matplotlib.ticker import LinearLocator, FormatStrFormatter\n",
|
||||
"import numpy as np\n",
|
||||
"from random import random, seed\n",
|
||||
"\n",
|
||||
"fig = plt.figure()\n",
|
||||
"ax = fig.gca(projection='3d')\n",
|
||||
"\n",
|
||||
"# Make data.\n",
|
||||
"x = np.arange(0, 1, 0.05)\n",
|
||||
"y = np.arange(0, 1, 0.05)\n",
|
||||
"x, y = np.meshgrid(x,y)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def FrankeFunction(x,y):\n",
|
||||
" term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))\n",
|
||||
" term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))\n",
|
||||
" term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))\n",
|
||||
" term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)\n",
|
||||
" return term1 + term2 + term3 + term4\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"z = FrankeFunction(x, y)\n",
|
||||
"\n",
|
||||
"# Plot the surface.\n",
|
||||
"surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,\n",
|
||||
" linewidth=0, antialiased=False)\n",
|
||||
"\n",
|
||||
"# Customize the z axis.\n",
|
||||
"ax.set_zlim(-0.10, 1.40)\n",
|
||||
"ax.zaxis.set_major_locator(LinearLocator(10))\n",
|
||||
"ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))\n",
|
||||
"\n",
|
||||
"# Add a color bar which maps values to colors.\n",
|
||||
"fig.colorbar(surf, shrink=0.5, aspect=5)\n",
|
||||
"\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Exercise: Ordinary Least Square (OLS) on the Franke function\n",
|
||||
"\n",
|
||||
"We will generate our own dataset for a function\n",
|
||||
"$\\mathrm{FrankeFunction}(x,y)$ with $x,y \\in [0,1]$. The function\n",
|
||||
"$f(x,y)$ is the Franke function. You should explore also the addition\n",
|
||||
"of an added stochastic noise to this function using the normal\n",
|
||||
"distribution $N(0,1)$.\n",
|
||||
"\n",
|
||||
"*Write your own code* (using either a matrix inversion or a singular\n",
|
||||
"value decomposition from e.g., **numpy** ) or use your code from\n",
|
||||
"homeworks 1 and 2 and perform a standard least square regression\n",
|
||||
"analysis using polynomials in $x$ and $y$ up to fifth order. Find the\n",
|
||||
"[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\beta$ by computing their\n",
|
||||
"variances, evaluate the Mean Squared error (MSE)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n",
|
||||
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"and the $R^2$ score function. If $\\tilde{\\hat{y}}_i$ is the predicted\n",
|
||||
"value of the $i-th$ sample and $y_i$ is the corresponding true value,\n",
|
||||
"then the score $R^2$ is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where we have defined the mean value of $\\hat{y}$ as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Your code has to include a scaling of the data (for example by\n",
|
||||
"subtracting the mean value), and\n",
|
||||
"a split of the data in training and test data. For this exercise you can\n",
|
||||
"either write your own code or use for example the function for\n",
|
||||
"splitting training data provided by the library **Scikit-Learn** (make\n",
|
||||
"sure you have installed it). This function is called\n",
|
||||
"$train\\_test\\_split$. **You should present a critical discussion of why and how you have scaled or not scaled the data**.\n",
|
||||
"\n",
|
||||
"It is normal in essentially all Machine Learning studies to split the\n",
|
||||
"data in a training set and a test set (eventually also an additional\n",
|
||||
"validation set). There\n",
|
||||
"is no explicit recipe for how much data should be included as training\n",
|
||||
"data and say test data. An accepted rule of thumb is to use\n",
|
||||
"approximately $2/3$ to $4/5$ of the data as training data.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"You can easily reuse the solutions to your exercises from week 35 and week 36.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Exercise: Bias-variance trade-off and resampling techniques\n",
|
||||
"\n",
|
||||
"Our aim here is to study the bias-variance trade-off by implementing the **bootstrap** resampling technique.\n",
|
||||
"\n",
|
||||
"With a code which does OLS and includes resampling techniques, \n",
|
||||
"we will now discuss the bias-variance trade-off in the context of\n",
|
||||
"continuous predictions such as regression. However, many of the\n",
|
||||
"intuitions and ideas discussed here also carry over to classification\n",
|
||||
"tasks and basically all Machine Learning algorithms. \n",
|
||||
"\n",
|
||||
"Before you perform an analysis of the bias-variance trade-off on your test data, make\n",
|
||||
"first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and\n",
|
||||
"Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to \n",
|
||||
"indicate possible regions of low/high bias and variance. You will most likely not get an\n",
|
||||
"equally smooth curve!\n",
|
||||
"\n",
|
||||
"With this result we move on to the bias-variance trade-off analysis.\n",
|
||||
"\n",
|
||||
"Consider a\n",
|
||||
"dataset $\\mathcal{L}$ consisting of the data\n",
|
||||
"$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n",
|
||||
"\n",
|
||||
"Let us assume that the true data is generated from a noisy model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{y}=f(\\boldsymbol{x}) + \\boldsymbol{\\epsilon}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here $\\epsilon$ is normally distributed with mean zero and standard\n",
|
||||
"deviation $\\sigma^2$.\n",
|
||||
"\n",
|
||||
"In our derivation of the ordinary least squares method we defined then\n",
|
||||
"an approximation to the function $f$ in terms of the parameters\n",
|
||||
"$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
|
||||
"that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n",
|
||||
"\n",
|
||||
"The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the means\n",
|
||||
"squared error via the so-called cost function"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here the expected value $\\mathbb{E}$ is the sample value. \n",
|
||||
"\n",
|
||||
"Show that you can rewrite this as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\frac{1}{n}\\sum_i(f_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\sigma^2.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Explain what the terms mean, which one is the bias and which one is\n",
|
||||
"the variance and discuss their interpretations.\n",
|
||||
"\n",
|
||||
"Perform then a bias-variance analysis of the Franke function by\n",
|
||||
"studying the MSE value as function of the complexity of your model.\n",
|
||||
"\n",
|
||||
"Discuss the bias and variance trade-off as function\n",
|
||||
"of your model complexity (the degree of the polynomial) and the number\n",
|
||||
"of data points, and possibly also your training and test data using the **bootstrap** resampling method.\n",
|
||||
"\n",
|
||||
"Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Exercise: Cross-validation as resampling techniques, adding more complexity\n",
|
||||
"\n",
|
||||
"The aim here is to write your own code for another widely popular\n",
|
||||
"resampling technique, the so-called cross-validation method. Again,\n",
|
||||
"before you start with cross-validation approach, you should scale your\n",
|
||||
"data.\n",
|
||||
"\n",
|
||||
"Implement the $k$-fold cross-validation algorithm (write your own\n",
|
||||
"code) and evaluate again the MSE function resulting\n",
|
||||
"from the test folds. You can compare your own code with that from\n",
|
||||
"**Scikit-Learn** if needed. \n",
|
||||
"\n",
|
||||
"Compare the MSE you get from your cross-validation code with the one\n",
|
||||
"you got from your **bootstrap** code. Comment your results. Try $5-10$\n",
|
||||
"folds. You can also compare your own cross-validation code with the\n",
|
||||
"one provided by **Scikit-Learn**.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Exercise: Ridge Regression on the Franke function with resampling\n",
|
||||
"\n",
|
||||
"Write your own code for the Ridge method, either using matrix\n",
|
||||
"inversion or the singular value decomposition as done in the previous\n",
|
||||
"exercise. Perform the same bootstrap analysis as in the\n",
|
||||
"Exercise 2 (for the same polynomials) and the cross-validation in exercise 3 but now for different values of $\\lambda$. Compare and\n",
|
||||
"analyze your results with those obtained in exercises 1-3. Study the\n",
|
||||
"dependence on $\\lambda$.\n",
|
||||
"\n",
|
||||
"Study also the bias-variance trade-off as function of various values of\n",
|
||||
"the parameter $\\lambda$. For the bias-variance trade-off, use the **bootstrap** resampling method. Comment your results. \n",
|
||||
"\n",
|
||||
"### Exercise: Lasso Regression on the Franke function with resampling\n",
|
||||
"\n",
|
||||
"This exercise is essentially a repeat of the previous two ones, but now\n",
|
||||
"with Lasso regression. Write either your own code (difficult and optional) or, in this case,\n",
|
||||
"you can also use the functionalities of **Scikit-Learn** (recommended). \n",
|
||||
"Give a\n",
|
||||
"critical discussion of the three methods and a judgement of which\n",
|
||||
"model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the **bootstrap** resampling technique and an analysis of the mean squared error using cross-validation. \n",
|
||||
"\n",
|
||||
"### Exercise: Analysis of real data\n",
|
||||
"\n",
|
||||
"With our codes functioning and having been tested properly on a\n",
|
||||
"simpler function we are now ready to look at real data. We will\n",
|
||||
"essentially repeat in this exercise what was done in exercises 1-5. However, we\n",
|
||||
"need first to download the data and prepare properly the inputs to our\n",
|
||||
"codes. We are going to download digital terrain data from the website\n",
|
||||
"<https://earthexplorer.usgs.gov/>,\n",
|
||||
"\n",
|
||||
"Or, if you prefer, we have placed selected datafiles at <https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2021/Project1/DataFiles>\n",
|
||||
"\n",
|
||||
"In order to obtain data for a specific region, you need to register as\n",
|
||||
"a user (free) at this website and then decide upon which area you want\n",
|
||||
"to fetch the digital terrain data from. In order to be able to read\n",
|
||||
"the data properly, you need to specify that the format should be **SRTM\n",
|
||||
"Arc-Second Global** and download the data as a **GeoTIF** file. The\n",
|
||||
"files are then stored in *tif* format which can be imported into a\n",
|
||||
"Python program using"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"scipy.misc.imread"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here is a simple part of a Python code which reads and plots the data\n",
|
||||
"from such files"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"from imageio import imread\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from mpl_toolkits.mplot3d import Axes3D\n",
|
||||
"from matplotlib import cm\n",
|
||||
"\n",
|
||||
"# Load the terrain\n",
|
||||
"terrain1 = imread('SRTM_data_Norway_1.tif')\n",
|
||||
"# Show the terrain\n",
|
||||
"plt.figure()\n",
|
||||
"plt.title('Terrain over Norway 1')\n",
|
||||
"plt.imshow(terrain1, cmap='gray')\n",
|
||||
"plt.xlabel('X')\n",
|
||||
"plt.ylabel('Y')\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If you should have problems in downloading the digital terrain data,\n",
|
||||
"we provide two examples under the data folder of project 1. One is\n",
|
||||
"from a region close to Stavanger in Norway and the other Møsvatn\n",
|
||||
"Austfjell, again in Norway.\n",
|
||||
"Feel free to produce your own terrain data.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example [kaggle.com](https://www.kaggle.com/datasets) for examples.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Our final part deals with the parameterization of your digital terrain\n",
|
||||
"data (or your own data). We will apply all three methods for linear regression, the same type (or higher order) of polynomial\n",
|
||||
"approximation and cross-validation as resampling technique to evaluate which\n",
|
||||
"model fits the data best.\n",
|
||||
"\n",
|
||||
"At the end, you should present a critical evaluation of your results\n",
|
||||
"and discuss the applicability of these regression methods to the type\n",
|
||||
"of data presented here (either the terrain data we propose or other data sets)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {},
|
||||
|
||||
@@ -52,8 +52,13 @@ For the reading assignments we use the following abbreviations:
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Resampling methods, cross-validation and Bootstrap
|
||||
- Lecture Friday: More on Resampling methods and summary of linear regression
|
||||
- Reading recommendations: See lecture notes for week 37 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Chapter
|
||||
- Reading recommendations:
|
||||
- Recommended Reading:
|
||||
- Lectures on Resampling methods for week 37 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Bishop 1.3 (cross-validation) and 3.2 (bias-variance tradeoff)
|
||||
- Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). This chapter is better than Bishop's on these topics. Goodfellow et al discuss some of these topics in sections 5.2-5.5.
|
||||
|
||||
|
||||
### Week 38 September 20-24
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Classification problems and Logistic Regression, from binary cases to several categories
|
||||
|
||||
@@ -321,6 +321,43 @@
|
||||
5.5. Cross-validation
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercises-and-projects">
|
||||
5.6. Exercises and Projects
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-ordinary-least-square-ols-on-the-franke-function">
|
||||
5.6.1. Exercise: Ordinary Least Square (OLS) on the Franke function
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-bias-variance-trade-off-and-resampling-techniques">
|
||||
5.6.2. Exercise: Bias-variance trade-off and resampling techniques
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-cross-validation-as-resampling-techniques-adding-more-complexity">
|
||||
5.6.3. Exercise: Cross-validation as resampling techniques, adding more complexity
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-ridge-regression-on-the-franke-function-with-resampling">
|
||||
5.6.4. Exercise: Ridge Regression on the Franke function with resampling
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-lasso-regression-on-the-franke-function-with-resampling">
|
||||
5.6.5. Exercise: Lasso Regression on the Franke function with resampling
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-analysis-of-real-data">
|
||||
5.6.6. Exercise: Analysis of real data
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -592,10 +629,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.1375 sec
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.137546 sec
|
||||
Jackknife Statistics :
|
||||
original bias std. error
|
||||
100.029 100.019 0.150581
|
||||
99.9031 99.8931 0.149233
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -841,14 +878,14 @@ Error: 0.32149601703519126
|
||||
Bias^2: 0.3123314713548606
|
||||
Var: 0.009164545680330616
|
||||
0.32149601703519126 >= 0.3123314713548606 + 0.009164545680330616 = 0.3214960170351912
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 1
|
||||
Polynomial degree: 1
|
||||
Error: 0.08426840630693411
|
||||
Bias^2: 0.07968918676726028
|
||||
Var: 0.004579219539673833
|
||||
0.08426840630693411 >= 0.07968918676726028 + 0.004579219539673833 = 0.08426840630693411
|
||||
Polynomial degree: 2
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 2
|
||||
Error: 0.10398646080125035
|
||||
Bias^2: 0.10077114273548986
|
||||
Var: 0.0032153180657605086
|
||||
@@ -868,21 +905,20 @@ Error: 0.05227921801205707
|
||||
Bias^2: 0.048187277304303125
|
||||
Var: 0.004091940707753964
|
||||
0.05227921801205707 >= 0.048187277304303125 + 0.004091940707753964 = 0.05227921801205709
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 6
|
||||
Polynomial degree: 6
|
||||
Error: 0.03781367141738898
|
||||
Bias^2: 0.03365768507152761
|
||||
Var: 0.004155986345861379
|
||||
0.03781367141738898 >= 0.03365768507152761 + 0.004155986345861379 = 0.03781367141738899
|
||||
Polynomial degree: 7
|
||||
Polynomial degree:
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 7
|
||||
Error: 0.027609773491022498
|
||||
Bias^2: 0.02299949826036597
|
||||
Var: 0.004610275230656537
|
||||
0.027609773491022498 >= 0.02299949826036597 + 0.004610275230656537 = 0.027609773491022505
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 8
|
||||
Polynomial degree: 8
|
||||
Error: 0.017355848195591973
|
||||
Bias^2: 0.010331721306655588
|
||||
Var: 0.007024126888936384
|
||||
@@ -892,7 +928,10 @@ Error: 0.026605727637189085
|
||||
Bias^2: 0.010018312644140933
|
||||
Var: 0.016587414993048166
|
||||
0.026605727637189085 >= 0.010018312644140933 + 0.016587414993048166 = 0.0266057276371891
|
||||
Polynomial degree: 10
|
||||
Polynomial degree:
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 10
|
||||
Error: 0.021592704588043153
|
||||
Bias^2: 0.010516485576652981
|
||||
Var: 0.011076219011390184
|
||||
@@ -902,23 +941,19 @@ Error: 0.07160048164228314
|
||||
Bias^2: 0.01443680008897583
|
||||
Var: 0.0571636815533073
|
||||
0.07160048164228314 >= 0.01443680008897583 + 0.0571636815533073 = 0.07160048164228312
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 12
|
||||
Polynomial degree: 12
|
||||
Error: 0.1154777721897675
|
||||
Bias^2: 0.01628578269590588
|
||||
Var: 0.09919198949386163
|
||||
0.1154777721897675 >= 0.01628578269590588 + 0.09919198949386163 = 0.11547777218976751
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 13
|
||||
Polynomial degree: 13
|
||||
Error: 0.22842468702166951
|
||||
Bias^2: 0.01975416527163567
|
||||
Var: 0.20867052175003387
|
||||
0.22842468702166951 >= 0.01975416527163567 + 0.20867052175003387 = 0.22842468702166954
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_38_6.png" src="_images/chapter3_38_6.png" />
|
||||
<img alt="_images/chapter3_38_4.png" src="_images/chapter3_38_4.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>The bias-variance tradeoff summarizes the fundamental tension in
|
||||
@@ -1139,11 +1174,11 @@ Mean squared error on test data: 123711.53703498
|
||||
Degree of polynomial: 3
|
||||
Mean squared error on training data: 9011.85263220
|
||||
Mean squared error on test data: 10913.84780262
|
||||
Degree of polynomial: 4
|
||||
Mean squared error on training data: 303.47610036
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error on test data: 426.30787294
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 4
|
||||
Mean squared error on training data: 303.47610036
|
||||
Mean squared error on test data: 426.30787294
|
||||
Degree of polynomial: 5
|
||||
Mean squared error on training data: 3.80354994
|
||||
Mean squared error on test data: 5.98822371
|
||||
@@ -1175,64 +1210,62 @@ Mean squared error on test data: 0.17446471
|
||||
Degree of polynomial: 13
|
||||
Mean squared error on training data: 0.00759119
|
||||
Mean squared error on test data: 1.08131003
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 14
|
||||
Degree of polynomial: 14
|
||||
Mean squared error on training data: 0.00472199
|
||||
Mean squared error on test data: 0.81333793
|
||||
Degree of polynomial: 15
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 15
|
||||
Mean squared error on training data: 0.00410478
|
||||
Mean squared error on test data: 92.09145189
|
||||
Degree of polynomial: 16
|
||||
Mean squared error on training data: 0.00315593
|
||||
Mean squared error on test data: 234.39716546
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 17
|
||||
Degree of polynomial: 17
|
||||
Mean squared error on training data: 0.00242998
|
||||
Mean squared error on test data: 1271.05295709
|
||||
Degree of polynomial: 18
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 18
|
||||
Mean squared error on training data: 0.00228740
|
||||
Mean squared error on test data: 108.42208194
|
||||
Degree of polynomial: 19
|
||||
Mean squared error on training data: 0.00156372
|
||||
Mean squared error on test data: 1388.41078073
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 20
|
||||
Degree of polynomial: 20
|
||||
Mean squared error on training data: 0.00137982
|
||||
Mean squared error on test data: 1761.43341615
|
||||
Degree of polynomial: 21
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 21
|
||||
Mean squared error on training data: 0.00118170
|
||||
Mean squared error on test data: 15061.31603087
|
||||
Degree of polynomial: 22
|
||||
Mean squared error on training data: 0.00092354
|
||||
Mean squared error on test data: 890.63488525
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 23
|
||||
Degree of polynomial: 23
|
||||
Mean squared error on training data: 0.00085887
|
||||
Mean squared error on test data: 5483.16796929
|
||||
Degree of polynomial: 24
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 24
|
||||
Mean squared error on training data: 0.00084589
|
||||
Mean squared error on test data: 1695.57143061
|
||||
Degree of polynomial: 25
|
||||
Mean squared error on training data: 0.00078806
|
||||
Mean squared error on test data: 131343.30655001
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 26
|
||||
Degree of polynomial: 26
|
||||
Mean squared error on training data: 0.00076916
|
||||
Mean squared error on test data: 17709.14370264
|
||||
Degree of polynomial: 27
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 27
|
||||
Mean squared error on training data: 0.00068970
|
||||
Mean squared error on test data: 2975.38903780
|
||||
Degree of polynomial: 28
|
||||
Mean squared error on training data: 0.00062588
|
||||
Mean squared error on test data: 3848.64522721
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 29
|
||||
Degree of polynomial: 29
|
||||
Mean squared error on training data: 0.00060728
|
||||
Mean squared error on test data: 2988.64001211
|
||||
</pre></div>
|
||||
@@ -1243,7 +1276,7 @@ Mean squared error on test data: 2988.64001211
|
||||
plt.plot(polynomial, np.log10(testerror), label='Test Error')
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_41_11.png" src="_images/chapter3_41_11.png" />
|
||||
<img alt="_images/chapter3_41_10.png" src="_images/chapter3_41_10.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1525,6 +1558,294 @@ cross-validation (LOOCV).</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="exercises-and-projects">
|
||||
<h2><span class="section-number">5.6. </span>Exercises and Projects<a class="headerlink" href="#exercises-and-projects" title="Permalink to this headline">¶</a></h2>
|
||||
<p>The main aim of this project is to study in more detail various
|
||||
regression methods, including the Ordinary Least Squares (OLS) method,
|
||||
The total score is <strong>100</strong> points. Each subtask has its own final score.</p>
|
||||
<p>We will first study how to fit polynomials to a specific
|
||||
two-dimensional function called <a class="reference external" href="http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf">Franke’s
|
||||
function</a>. This
|
||||
is a function which has been widely used when testing various
|
||||
interpolation and fitting algorithms. Furthermore, after having
|
||||
established the model and the method, we will employ resamling
|
||||
techniques such as cross-validation and/or bootstrap in order to perform a
|
||||
proper assessment of our models. We will also study in detail the
|
||||
so-called Bias-Variance trade off.</p>
|
||||
<p>The Franke function, which is a weighted sum of four exponentials reads as follows</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\begin{align*}
|
||||
f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\
|
||||
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
|
||||
\end{align*}
|
||||
\end{split}\]</div>
|
||||
<p>The function will be defined for <span class="math notranslate nohighlight">\(x,y\in [0,1]\)</span>. Our first step will
|
||||
be to perform an OLS regression analysis of this function, trying out
|
||||
a polynomial fit with an <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> dependence of the form <span class="math notranslate nohighlight">\([x, y,
|
||||
x^2, y^2, xy, \dots]\)</span>. We will also include bootstrap first as
|
||||
a resampling technique. After that we will include the cross-validation technique. As in homeworks 1 and 2, we can use a uniform
|
||||
distribution to set up the arrays of values for <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span>, or as in
|
||||
the example below just a set of fixed
|
||||
values for <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> with a given step
|
||||
size. We will fit a
|
||||
function (for example a polynomial) of <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span>. Thereafter we
|
||||
will repeat much of the same procedure using the Ridge and Lasso
|
||||
regression methods, introducing thus a dependence on the bias
|
||||
(penalty) <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>Finally we are going to use (real) digital terrain data and try to
|
||||
reproduce these data using the same methods. We will also try to go
|
||||
beyond the second-order polynomials metioned above and explore
|
||||
which polynomial fits the data best.</p>
|
||||
<p>The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">mpl_toolkits.mplot3d</span> <span class="kn">import</span> <span class="n">Axes3D</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">matplotlib</span> <span class="kn">import</span> <span class="n">cm</span>
|
||||
<span class="kn">from</span> <span class="nn">matplotlib.ticker</span> <span class="kn">import</span> <span class="n">LinearLocator</span><span class="p">,</span> <span class="n">FormatStrFormatter</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
|
||||
|
||||
<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
|
||||
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">gca</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">'3d'</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Make data.</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mf">0.05</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mf">0.05</span><span class="p">)</span>
|
||||
<span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">meshgrid</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">):</span>
|
||||
<span class="n">term1</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mf">0.25</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
|
||||
<span class="n">term2</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="mf">49.0</span> <span class="o">-</span> <span class="mf">0.1</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">+</span><span class="mi">1</span><span class="p">))</span>
|
||||
<span class="n">term3</span> <span class="o">=</span> <span class="mf">0.5</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="o">/</span><span class="mf">4.0</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">3</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
|
||||
<span class="n">term4</span> <span class="o">=</span> <span class="o">-</span><span class="mf">0.2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">4</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span> <span class="o">-</span> <span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">term1</span> <span class="o">+</span> <span class="n">term2</span> <span class="o">+</span> <span class="n">term3</span> <span class="o">+</span> <span class="n">term4</span>
|
||||
|
||||
|
||||
<span class="n">z</span> <span class="o">=</span> <span class="n">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Plot the surface.</span>
|
||||
<span class="n">surf</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">z</span><span class="p">,</span> <span class="n">cmap</span><span class="o">=</span><span class="n">cm</span><span class="o">.</span><span class="n">coolwarm</span><span class="p">,</span>
|
||||
<span class="n">linewidth</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">antialiased</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Customize the z axis.</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">set_zlim</span><span class="p">(</span><span class="o">-</span><span class="mf">0.10</span><span class="p">,</span> <span class="mf">1.40</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">zaxis</span><span class="o">.</span><span class="n">set_major_locator</span><span class="p">(</span><span class="n">LinearLocator</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">zaxis</span><span class="o">.</span><span class="n">set_major_formatter</span><span class="p">(</span><span class="n">FormatStrFormatter</span><span class="p">(</span><span class="s1">'</span><span class="si">%.02f</span><span class="s1">'</span><span class="p">))</span>
|
||||
|
||||
<span class="c1"># Add a color bar which maps values to colors.</span>
|
||||
<span class="n">fig</span><span class="o">.</span><span class="n">colorbar</span><span class="p">(</span><span class="n">surf</span><span class="p">,</span> <span class="n">shrink</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">aspect</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter3_54_0.png" src="_images/chapter3_54_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="exercise-ordinary-least-square-ols-on-the-franke-function">
|
||||
<h3><span class="section-number">5.6.1. </span>Exercise: Ordinary Least Square (OLS) on the Franke function<a class="headerlink" href="#exercise-ordinary-least-square-ols-on-the-franke-function" title="Permalink to this headline">¶</a></h3>
|
||||
<p>We will generate our own dataset for a function
|
||||
<span class="math notranslate nohighlight">\(\mathrm{FrankeFunction}(x,y)\)</span> with <span class="math notranslate nohighlight">\(x,y \in [0,1]\)</span>. The function
|
||||
<span class="math notranslate nohighlight">\(f(x,y)\)</span> is the Franke function. You should explore also the addition
|
||||
of an added stochastic noise to this function using the normal
|
||||
distribution <span class="math notranslate nohighlight">\(N(0,1)\)</span>.</p>
|
||||
<p><em>Write your own code</em> (using either a matrix inversion or a singular
|
||||
value decomposition from e.g., <strong>numpy</strong> ) or use your code from
|
||||
homeworks 1 and 2 and perform a standard least square regression
|
||||
analysis using polynomials in <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> up to fifth order. Find the
|
||||
<a class="reference external" href="https://en.wikipedia.org/wiki/Confidence_interval">confidence intervals</a> of the parameters (estimators) <span class="math notranslate nohighlight">\(\beta\)</span> by computing their
|
||||
variances, evaluate the Mean Squared error (MSE)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
\]</div>
|
||||
<p>and the <span class="math notranslate nohighlight">\(R^2\)</span> score function. If <span class="math notranslate nohighlight">\(\tilde{\hat{y}}_i\)</span> is the predicted
|
||||
value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value,
|
||||
then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
\]</div>
|
||||
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\hat{y}\)</span> as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
\]</div>
|
||||
<p>Your code has to include a scaling of the data (for example by
|
||||
subtracting the mean value), and
|
||||
a split of the data in training and test data. For this exercise you can
|
||||
either write your own code or use for example the function for
|
||||
splitting training data provided by the library <strong>Scikit-Learn</strong> (make
|
||||
sure you have installed it). This function is called
|
||||
<span class="math notranslate nohighlight">\(train\_test\_split\)</span>. <strong>You should present a critical discussion of why and how you have scaled or not scaled the data</strong>.</p>
|
||||
<p>It is normal in essentially all Machine Learning studies to split the
|
||||
data in a training set and a test set (eventually also an additional
|
||||
validation set). There
|
||||
is no explicit recipe for how much data should be included as training
|
||||
data and say test data. An accepted rule of thumb is to use
|
||||
approximately <span class="math notranslate nohighlight">\(2/3\)</span> to <span class="math notranslate nohighlight">\(4/5\)</span> of the data as training data.</p>
|
||||
<p>You can easily reuse the solutions to your exercises from week 35 and week 36.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-bias-variance-trade-off-and-resampling-techniques">
|
||||
<h3><span class="section-number">5.6.2. </span>Exercise: Bias-variance trade-off and resampling techniques<a class="headerlink" href="#exercise-bias-variance-trade-off-and-resampling-techniques" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Our aim here is to study the bias-variance trade-off by implementing the <strong>bootstrap</strong> resampling technique.</p>
|
||||
<p>With a code which does OLS and includes resampling techniques,
|
||||
we will now discuss the bias-variance trade-off in the context of
|
||||
continuous predictions such as regression. However, many of the
|
||||
intuitions and ideas discussed here also carry over to classification
|
||||
tasks and basically all Machine Learning algorithms.</p>
|
||||
<p>Before you perform an analysis of the bias-variance trade-off on your test data, make
|
||||
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
|
||||
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
|
||||
indicate possible regions of low/high bias and variance. You will most likely not get an
|
||||
equally smooth curve!</p>
|
||||
<p>With this result we move on to the bias-variance trade-off analysis.</p>
|
||||
<p>Consider a
|
||||
dataset <span class="math notranslate nohighlight">\(\mathcal{L}\)</span> consisting of the data
|
||||
<span class="math notranslate nohighlight">\(\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}\)</span>.</p>
|
||||
<p>Let us assume that the true data is generated from a noisy model</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
|
||||
\]</div>
|
||||
<p>Here <span class="math notranslate nohighlight">\(\epsilon\)</span> is normally distributed with mean zero and standard
|
||||
deviation <span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
|
||||
<p>In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function <span class="math notranslate nohighlight">\(f\)</span> in terms of the parameters
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
|
||||
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>The parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> are in turn found by optimizing the means
|
||||
squared error via the so-called cost function</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
\]</div>
|
||||
<p>Here the expected value <span class="math notranslate nohighlight">\(\mathbb{E}\)</span> is the sample value.</p>
|
||||
<p>Show that you can rewrite this as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2.
|
||||
\]</div>
|
||||
<p>Explain what the terms mean, which one is the bias and which one is
|
||||
the variance and discuss their interpretations.</p>
|
||||
<p>Perform then a bias-variance analysis of the Franke function by
|
||||
studying the MSE value as function of the complexity of your model.</p>
|
||||
<p>Discuss the bias and variance trade-off as function
|
||||
of your model complexity (the degree of the polynomial) and the number
|
||||
of data points, and possibly also your training and test data using the <strong>bootstrap</strong> resampling method.</p>
|
||||
<p>Note also that when you calculate the bias, in all applications you don’t know the function values <span class="math notranslate nohighlight">\(f_i\)</span>. You would hence replace them with the actual data points <span class="math notranslate nohighlight">\(y_i\)</span>.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-cross-validation-as-resampling-techniques-adding-more-complexity">
|
||||
<h3><span class="section-number">5.6.3. </span>Exercise: Cross-validation as resampling techniques, adding more complexity<a class="headerlink" href="#exercise-cross-validation-as-resampling-techniques-adding-more-complexity" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The aim here is to write your own code for another widely popular
|
||||
resampling technique, the so-called cross-validation method. Again,
|
||||
before you start with cross-validation approach, you should scale your
|
||||
data.</p>
|
||||
<p>Implement the <span class="math notranslate nohighlight">\(k\)</span>-fold cross-validation algorithm (write your own
|
||||
code) and evaluate again the MSE function resulting
|
||||
from the test folds. You can compare your own code with that from
|
||||
<strong>Scikit-Learn</strong> if needed.</p>
|
||||
<p>Compare the MSE you get from your cross-validation code with the one
|
||||
you got from your <strong>bootstrap</strong> code. Comment your results. Try <span class="math notranslate nohighlight">\(5-10\)</span>
|
||||
folds. You can also compare your own cross-validation code with the
|
||||
one provided by <strong>Scikit-Learn</strong>.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-ridge-regression-on-the-franke-function-with-resampling">
|
||||
<h3><span class="section-number">5.6.4. </span>Exercise: Ridge Regression on the Franke function with resampling<a class="headerlink" href="#exercise-ridge-regression-on-the-franke-function-with-resampling" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Write your own code for the Ridge method, either using matrix
|
||||
inversion or the singular value decomposition as done in the previous
|
||||
exercise. Perform the same bootstrap analysis as in the
|
||||
Exercise 2 (for the same polynomials) and the cross-validation in exercise 3 but now for different values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Compare and
|
||||
analyze your results with those obtained in exercises 1-3. Study the
|
||||
dependence on <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>Study also the bias-variance trade-off as function of various values of
|
||||
the parameter <span class="math notranslate nohighlight">\(\lambda\)</span>. For the bias-variance trade-off, use the <strong>bootstrap</strong> resampling method. Comment your results.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-lasso-regression-on-the-franke-function-with-resampling">
|
||||
<h3><span class="section-number">5.6.5. </span>Exercise: Lasso Regression on the Franke function with resampling<a class="headerlink" href="#exercise-lasso-regression-on-the-franke-function-with-resampling" title="Permalink to this headline">¶</a></h3>
|
||||
<p>This exercise is essentially a repeat of the previous two ones, but now
|
||||
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
|
||||
you can also use the functionalities of <strong>Scikit-Learn</strong> (recommended).
|
||||
Give a
|
||||
critical discussion of the three methods and a judgement of which
|
||||
model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the <strong>bootstrap</strong> resampling technique and an analysis of the mean squared error using cross-validation.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-analysis-of-real-data">
|
||||
<h3><span class="section-number">5.6.6. </span>Exercise: Analysis of real data<a class="headerlink" href="#exercise-analysis-of-real-data" title="Permalink to this headline">¶</a></h3>
|
||||
<p>With our codes functioning and having been tested properly on a
|
||||
simpler function we are now ready to look at real data. We will
|
||||
essentially repeat in this exercise what was done in exercises 1-5. However, we
|
||||
need first to download the data and prepare properly the inputs to our
|
||||
codes. We are going to download digital terrain data from the website
|
||||
<a class="reference external" href="https://earthexplorer.usgs.gov/">https://earthexplorer.usgs.gov/</a>,</p>
|
||||
<p>Or, if you prefer, we have placed selected datafiles at <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2021/Project1/DataFiles">https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2021/Project1/DataFiles</a></p>
|
||||
<p>In order to obtain data for a specific region, you need to register as
|
||||
a user (free) at this website and then decide upon which area you want
|
||||
to fetch the digital terrain data from. In order to be able to read
|
||||
the data properly, you need to specify that the format should be <strong>SRTM
|
||||
Arc-Second Global</strong> and download the data as a <strong>GeoTIF</strong> file. The
|
||||
files are then stored in <em>tif</em> format which can be imported into a
|
||||
Python program using</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">scipy</span><span class="o">.</span><span class="n">misc</span><span class="o">.</span><span class="n">imread</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
<span class="ne">NameError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
|
||||
<span class="o"><</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="n">d985fb40c43d</span><span class="o">></span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
|
||||
<span class="ne">----> </span><span class="mi">1</span> <span class="n">scipy</span><span class="o">.</span><span class="n">misc</span><span class="o">.</span><span class="n">imread</span>
|
||||
|
||||
<span class="ne">NameError</span>: name 'scipy' is not defined
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Here is a simple part of a Python code which reads and plots the data
|
||||
from such files</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">imageio</span> <span class="kn">import</span> <span class="n">imread</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">mpl_toolkits.mplot3d</span> <span class="kn">import</span> <span class="n">Axes3D</span>
|
||||
<span class="kn">from</span> <span class="nn">matplotlib</span> <span class="kn">import</span> <span class="n">cm</span>
|
||||
|
||||
<span class="c1"># Load the terrain</span>
|
||||
<span class="n">terrain1</span> <span class="o">=</span> <span class="n">imread</span><span class="p">(</span><span class="s1">'SRTM_data_Norway_1.tif'</span><span class="p">)</span>
|
||||
<span class="c1"># Show the terrain</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s1">'Terrain over Norway 1'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">terrain1</span><span class="p">,</span> <span class="n">cmap</span><span class="o">=</span><span class="s1">'gray'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">'X'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">'Y'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>If you should have problems in downloading the digital terrain data,
|
||||
we provide two examples under the data folder of project 1. One is
|
||||
from a region close to Stavanger in Norway and the other Møsvatn
|
||||
Austfjell, again in Norway.
|
||||
Feel free to produce your own terrain data.</p>
|
||||
<p>Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example <a class="reference external" href="https://www.kaggle.com/datasets">kaggle.com</a> for examples.</p>
|
||||
<p>Our final part deals with the parameterization of your digital terrain
|
||||
data (or your own data). We will apply all three methods for linear regression, the same type (or higher order) of polynomial
|
||||
approximation and cross-validation as resampling technique to evaluate which
|
||||
model fits the data best.</p>
|
||||
<p>At the end, you should present a critical evaluation of your results
|
||||
and discuss the applicability of these regression methods to the type
|
||||
of data presented here (either the terrain data we propose or other data sets).</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script type="text/x-thebe-config">
|
||||
|
||||
@@ -443,9 +443,12 @@
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Resampling methods, cross-validation and Bootstrap</p></li>
|
||||
<li><p>Lecture Friday: More on Resampling methods and summary of linear regression</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 37 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>.</p>
|
||||
<li><p>Reading recommendations:</p></li>
|
||||
<li><p>Recommended Reading:</p>
|
||||
<ul>
|
||||
<li><p>Chapter</p></li>
|
||||
<li><p>Lectures on Resampling methods for week 37 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>.</p></li>
|
||||
<li><p>Bishop 1.3 (cross-validation) and 3.2 (bias-variance tradeoff)</p></li>
|
||||
<li><p>Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). This chapter is better than Bishop’s on these topics. Goodfellow et al discuss some of these topics in sections 5.2-5.5.</p></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -984,4 +984,311 @@ plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')
|
||||
plt.xlabel('log10(lambda)')
|
||||
plt.ylabel('MSE')
|
||||
plt.legend()
|
||||
plt.show()
|
||||
plt.show()
|
||||
|
||||
## Exercises and Projects
|
||||
|
||||
|
||||
|
||||
The main aim of this project is to study in more detail various
|
||||
regression methods, including the Ordinary Least Squares (OLS) method,
|
||||
The total score is **100** points. Each subtask has its own final score.
|
||||
|
||||
|
||||
We will first study how to fit polynomials to a specific
|
||||
two-dimensional function called [Franke's
|
||||
function](http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf). This
|
||||
is a function which has been widely used when testing various
|
||||
interpolation and fitting algorithms. Furthermore, after having
|
||||
established the model and the method, we will employ resamling
|
||||
techniques such as cross-validation and/or bootstrap in order to perform a
|
||||
proper assessment of our models. We will also study in detail the
|
||||
so-called Bias-Variance trade off.
|
||||
|
||||
|
||||
The Franke function, which is a weighted sum of four exponentials reads as follows
|
||||
|
||||
$$
|
||||
\begin{align*}
|
||||
f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\
|
||||
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
The function will be defined for $x,y\in [0,1]$. Our first step will
|
||||
be to perform an OLS regression analysis of this function, trying out
|
||||
a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,
|
||||
x^2, y^2, xy, \dots]$. We will also include bootstrap first as
|
||||
a resampling technique. After that we will include the cross-validation technique. As in homeworks 1 and 2, we can use a uniform
|
||||
distribution to set up the arrays of values for $x$ and $y$, or as in
|
||||
the example below just a set of fixed
|
||||
values for $x$ and $y$ with a given step
|
||||
size. We will fit a
|
||||
function (for example a polynomial) of $x$ and $y$. Thereafter we
|
||||
will repeat much of the same procedure using the Ridge and Lasso
|
||||
regression methods, introducing thus a dependence on the bias
|
||||
(penalty) $\lambda$.
|
||||
|
||||
Finally we are going to use (real) digital terrain data and try to
|
||||
reproduce these data using the same methods. We will also try to go
|
||||
beyond the second-order polynomials metioned above and explore
|
||||
which polynomial fits the data best.
|
||||
|
||||
|
||||
The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)
|
||||
|
||||
from mpl_toolkits.mplot3d import Axes3D
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib import cm
|
||||
from matplotlib.ticker import LinearLocator, FormatStrFormatter
|
||||
import numpy as np
|
||||
from random import random, seed
|
||||
|
||||
fig = plt.figure()
|
||||
ax = fig.gca(projection='3d')
|
||||
|
||||
# Make data.
|
||||
x = np.arange(0, 1, 0.05)
|
||||
y = np.arange(0, 1, 0.05)
|
||||
x, y = np.meshgrid(x,y)
|
||||
|
||||
|
||||
def FrankeFunction(x,y):
|
||||
term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
|
||||
term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
|
||||
term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
|
||||
term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
|
||||
return term1 + term2 + term3 + term4
|
||||
|
||||
|
||||
z = FrankeFunction(x, y)
|
||||
|
||||
# Plot the surface.
|
||||
surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,
|
||||
linewidth=0, antialiased=False)
|
||||
|
||||
# Customize the z axis.
|
||||
ax.set_zlim(-0.10, 1.40)
|
||||
ax.zaxis.set_major_locator(LinearLocator(10))
|
||||
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
|
||||
|
||||
# Add a color bar which maps values to colors.
|
||||
fig.colorbar(surf, shrink=0.5, aspect=5)
|
||||
|
||||
plt.show()
|
||||
|
||||
### Exercise: Ordinary Least Square (OLS) on the Franke function
|
||||
|
||||
We will generate our own dataset for a function
|
||||
$\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function
|
||||
$f(x,y)$ is the Franke function. You should explore also the addition
|
||||
of an added stochastic noise to this function using the normal
|
||||
distribution $N(0,1)$.
|
||||
|
||||
*Write your own code* (using either a matrix inversion or a singular
|
||||
value decomposition from e.g., **numpy** ) or use your code from
|
||||
homeworks 1 and 2 and perform a standard least square regression
|
||||
analysis using polynomials in $x$ and $y$ up to fifth order. Find the
|
||||
[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\beta$ by computing their
|
||||
variances, evaluate the Mean Squared error (MSE)
|
||||
|
||||
$$
|
||||
MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
$$
|
||||
|
||||
and the $R^2$ score function. If $\tilde{\hat{y}}_i$ is the predicted
|
||||
value of the $i-th$ sample and $y_i$ is the corresponding true value,
|
||||
then the score $R^2$ is defined as
|
||||
|
||||
$$
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
$$
|
||||
|
||||
where we have defined the mean value of $\hat{y}$ as
|
||||
|
||||
$$
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
$$
|
||||
|
||||
Your code has to include a scaling of the data (for example by
|
||||
subtracting the mean value), and
|
||||
a split of the data in training and test data. For this exercise you can
|
||||
either write your own code or use for example the function for
|
||||
splitting training data provided by the library **Scikit-Learn** (make
|
||||
sure you have installed it). This function is called
|
||||
$train\_test\_split$. **You should present a critical discussion of why and how you have scaled or not scaled the data**.
|
||||
|
||||
It is normal in essentially all Machine Learning studies to split the
|
||||
data in a training set and a test set (eventually also an additional
|
||||
validation set). There
|
||||
is no explicit recipe for how much data should be included as training
|
||||
data and say test data. An accepted rule of thumb is to use
|
||||
approximately $2/3$ to $4/5$ of the data as training data.
|
||||
|
||||
|
||||
You can easily reuse the solutions to your exercises from week 35 and week 36.
|
||||
|
||||
|
||||
|
||||
### Exercise: Bias-variance trade-off and resampling techniques
|
||||
|
||||
Our aim here is to study the bias-variance trade-off by implementing the **bootstrap** resampling technique.
|
||||
|
||||
With a code which does OLS and includes resampling techniques,
|
||||
we will now discuss the bias-variance trade-off in the context of
|
||||
continuous predictions such as regression. However, many of the
|
||||
intuitions and ideas discussed here also carry over to classification
|
||||
tasks and basically all Machine Learning algorithms.
|
||||
|
||||
Before you perform an analysis of the bias-variance trade-off on your test data, make
|
||||
first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and
|
||||
Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to
|
||||
indicate possible regions of low/high bias and variance. You will most likely not get an
|
||||
equally smooth curve!
|
||||
|
||||
With this result we move on to the bias-variance trade-off analysis.
|
||||
|
||||
Consider a
|
||||
dataset $\mathcal{L}$ consisting of the data
|
||||
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
|
||||
|
||||
Let us assume that the true data is generated from a noisy model
|
||||
|
||||
$$
|
||||
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
|
||||
$$
|
||||
|
||||
Here $\epsilon$ is normally distributed with mean zero and standard
|
||||
deviation $\sigma^2$.
|
||||
|
||||
In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function $f$ in terms of the parameters
|
||||
$\boldsymbol{\beta}$ and the design matrix $\boldsymbol{X}$ which embody our model,
|
||||
that is $\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}$.
|
||||
|
||||
The parameters $\boldsymbol{\beta}$ are in turn found by optimizing the means
|
||||
squared error via the so-called cost function
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
$$
|
||||
|
||||
Here the expected value $\mathbb{E}$ is the sample value.
|
||||
|
||||
Show that you can rewrite this as
|
||||
|
||||
$$
|
||||
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2.
|
||||
$$
|
||||
|
||||
Explain what the terms mean, which one is the bias and which one is
|
||||
the variance and discuss their interpretations.
|
||||
|
||||
Perform then a bias-variance analysis of the Franke function by
|
||||
studying the MSE value as function of the complexity of your model.
|
||||
|
||||
Discuss the bias and variance trade-off as function
|
||||
of your model complexity (the degree of the polynomial) and the number
|
||||
of data points, and possibly also your training and test data using the **bootstrap** resampling method.
|
||||
|
||||
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
|
||||
|
||||
|
||||
### Exercise: Cross-validation as resampling techniques, adding more complexity
|
||||
|
||||
The aim here is to write your own code for another widely popular
|
||||
resampling technique, the so-called cross-validation method. Again,
|
||||
before you start with cross-validation approach, you should scale your
|
||||
data.
|
||||
|
||||
Implement the $k$-fold cross-validation algorithm (write your own
|
||||
code) and evaluate again the MSE function resulting
|
||||
from the test folds. You can compare your own code with that from
|
||||
**Scikit-Learn** if needed.
|
||||
|
||||
Compare the MSE you get from your cross-validation code with the one
|
||||
you got from your **bootstrap** code. Comment your results. Try $5-10$
|
||||
folds. You can also compare your own cross-validation code with the
|
||||
one provided by **Scikit-Learn**.
|
||||
|
||||
|
||||
### Exercise: Ridge Regression on the Franke function with resampling
|
||||
|
||||
Write your own code for the Ridge method, either using matrix
|
||||
inversion or the singular value decomposition as done in the previous
|
||||
exercise. Perform the same bootstrap analysis as in the
|
||||
Exercise 2 (for the same polynomials) and the cross-validation in exercise 3 but now for different values of $\lambda$. Compare and
|
||||
analyze your results with those obtained in exercises 1-3. Study the
|
||||
dependence on $\lambda$.
|
||||
|
||||
Study also the bias-variance trade-off as function of various values of
|
||||
the parameter $\lambda$. For the bias-variance trade-off, use the **bootstrap** resampling method. Comment your results.
|
||||
|
||||
### Exercise: Lasso Regression on the Franke function with resampling
|
||||
|
||||
This exercise is essentially a repeat of the previous two ones, but now
|
||||
with Lasso regression. Write either your own code (difficult and optional) or, in this case,
|
||||
you can also use the functionalities of **Scikit-Learn** (recommended).
|
||||
Give a
|
||||
critical discussion of the three methods and a judgement of which
|
||||
model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the **bootstrap** resampling technique and an analysis of the mean squared error using cross-validation.
|
||||
|
||||
### Exercise: Analysis of real data
|
||||
|
||||
With our codes functioning and having been tested properly on a
|
||||
simpler function we are now ready to look at real data. We will
|
||||
essentially repeat in this exercise what was done in exercises 1-5. However, we
|
||||
need first to download the data and prepare properly the inputs to our
|
||||
codes. We are going to download digital terrain data from the website
|
||||
<https://earthexplorer.usgs.gov/>,
|
||||
|
||||
Or, if you prefer, we have placed selected datafiles at <https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2021/Project1/DataFiles>
|
||||
|
||||
In order to obtain data for a specific region, you need to register as
|
||||
a user (free) at this website and then decide upon which area you want
|
||||
to fetch the digital terrain data from. In order to be able to read
|
||||
the data properly, you need to specify that the format should be **SRTM
|
||||
Arc-Second Global** and download the data as a **GeoTIF** file. The
|
||||
files are then stored in *tif* format which can be imported into a
|
||||
Python program using
|
||||
|
||||
scipy.misc.imread
|
||||
|
||||
Here is a simple part of a Python code which reads and plots the data
|
||||
from such files
|
||||
|
||||
import numpy as np
|
||||
from imageio import imread
|
||||
import matplotlib.pyplot as plt
|
||||
from mpl_toolkits.mplot3d import Axes3D
|
||||
from matplotlib import cm
|
||||
|
||||
# Load the terrain
|
||||
terrain1 = imread('SRTM_data_Norway_1.tif')
|
||||
# Show the terrain
|
||||
plt.figure()
|
||||
plt.title('Terrain over Norway 1')
|
||||
plt.imshow(terrain1, cmap='gray')
|
||||
plt.xlabel('X')
|
||||
plt.ylabel('Y')
|
||||
plt.show()
|
||||
|
||||
If you should have problems in downloading the digital terrain data,
|
||||
we provide two examples under the data folder of project 1. One is
|
||||
from a region close to Stavanger in Norway and the other Møsvatn
|
||||
Austfjell, again in Norway.
|
||||
Feel free to produce your own terrain data.
|
||||
|
||||
|
||||
Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example [kaggle.com](https://www.kaggle.com/datasets) for examples.
|
||||
|
||||
|
||||
Our final part deals with the parameterization of your digital terrain
|
||||
data (or your own data). We will apply all three methods for linear regression, the same type (or higher order) of polynomial
|
||||
approximation and cross-validation as resampling technique to evaluate which
|
||||
model fits the data best.
|
||||
|
||||
At the end, you should present a critical evaluation of your results
|
||||
and discuss the applicability of these regression methods to the type
|
||||
of data presented here (either the terrain data we propose or other data sets).
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
@@ -1282,6 +1282,433 @@
|
||||
"plt.legend()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercises and Projects\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The main aim of this project is to study in more detail various\n",
|
||||
"regression methods, including the Ordinary Least Squares (OLS) method,\n",
|
||||
"The total score is **100** points. Each subtask has its own final score.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"We will first study how to fit polynomials to a specific\n",
|
||||
"two-dimensional function called [Franke's\n",
|
||||
"function](http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf). This\n",
|
||||
"is a function which has been widely used when testing various\n",
|
||||
"interpolation and fitting algorithms. Furthermore, after having\n",
|
||||
"established the model and the method, we will employ resamling\n",
|
||||
"techniques such as cross-validation and/or bootstrap in order to perform a\n",
|
||||
"proper assessment of our models. We will also study in detail the\n",
|
||||
"so-called Bias-Variance trade off.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The Franke function, which is a weighted sum of four exponentials reads as follows"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{align*}\n",
|
||||
"f(x,y) &= \\frac{3}{4}\\exp{\\left(-\\frac{(9x-2)^2}{4} - \\frac{(9y-2)^2}{4}\\right)}+\\frac{3}{4}\\exp{\\left(-\\frac{(9x+1)^2}{49}- \\frac{(9y+1)}{10}\\right)} \\\\\n",
|
||||
"&+\\frac{1}{2}\\exp{\\left(-\\frac{(9x-7)^2}{4} - \\frac{(9y-3)^2}{4}\\right)} -\\frac{1}{5}\\exp{\\left(-(9x-4)^2 - (9y-7)^2\\right) }.\n",
|
||||
"\\end{align*}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The function will be defined for $x,y\\in [0,1]$. Our first step will\n",
|
||||
"be to perform an OLS regression analysis of this function, trying out\n",
|
||||
"a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,\n",
|
||||
"x^2, y^2, xy, \\dots]$. We will also include bootstrap first as\n",
|
||||
"a resampling technique. After that we will include the cross-validation technique. As in homeworks 1 and 2, we can use a uniform\n",
|
||||
"distribution to set up the arrays of values for $x$ and $y$, or as in\n",
|
||||
"the example below just a set of fixed \n",
|
||||
"values for $x$ and $y$ with a given step\n",
|
||||
"size. We will fit a\n",
|
||||
"function (for example a polynomial) of $x$ and $y$. Thereafter we\n",
|
||||
"will repeat much of the same procedure using the Ridge and Lasso\n",
|
||||
"regression methods, introducing thus a dependence on the bias\n",
|
||||
"(penalty) $\\lambda$.\n",
|
||||
"\n",
|
||||
"Finally we are going to use (real) digital terrain data and try to\n",
|
||||
"reproduce these data using the same methods. We will also try to go\n",
|
||||
"beyond the second-order polynomials metioned above and explore \n",
|
||||
"which polynomial fits the data best.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The Python code for the Franke function is included here (it performs also a three-dimensional plot of it)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from mpl_toolkits.mplot3d import Axes3D\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from matplotlib import cm\n",
|
||||
"from matplotlib.ticker import LinearLocator, FormatStrFormatter\n",
|
||||
"import numpy as np\n",
|
||||
"from random import random, seed\n",
|
||||
"\n",
|
||||
"fig = plt.figure()\n",
|
||||
"ax = fig.gca(projection='3d')\n",
|
||||
"\n",
|
||||
"# Make data.\n",
|
||||
"x = np.arange(0, 1, 0.05)\n",
|
||||
"y = np.arange(0, 1, 0.05)\n",
|
||||
"x, y = np.meshgrid(x,y)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def FrankeFunction(x,y):\n",
|
||||
" term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))\n",
|
||||
" term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))\n",
|
||||
" term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))\n",
|
||||
" term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)\n",
|
||||
" return term1 + term2 + term3 + term4\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"z = FrankeFunction(x, y)\n",
|
||||
"\n",
|
||||
"# Plot the surface.\n",
|
||||
"surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,\n",
|
||||
" linewidth=0, antialiased=False)\n",
|
||||
"\n",
|
||||
"# Customize the z axis.\n",
|
||||
"ax.set_zlim(-0.10, 1.40)\n",
|
||||
"ax.zaxis.set_major_locator(LinearLocator(10))\n",
|
||||
"ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))\n",
|
||||
"\n",
|
||||
"# Add a color bar which maps values to colors.\n",
|
||||
"fig.colorbar(surf, shrink=0.5, aspect=5)\n",
|
||||
"\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Exercise: Ordinary Least Square (OLS) on the Franke function\n",
|
||||
"\n",
|
||||
"We will generate our own dataset for a function\n",
|
||||
"$\\mathrm{FrankeFunction}(x,y)$ with $x,y \\in [0,1]$. The function\n",
|
||||
"$f(x,y)$ is the Franke function. You should explore also the addition\n",
|
||||
"of an added stochastic noise to this function using the normal\n",
|
||||
"distribution $N(0,1)$.\n",
|
||||
"\n",
|
||||
"*Write your own code* (using either a matrix inversion or a singular\n",
|
||||
"value decomposition from e.g., **numpy** ) or use your code from\n",
|
||||
"homeworks 1 and 2 and perform a standard least square regression\n",
|
||||
"analysis using polynomials in $x$ and $y$ up to fifth order. Find the\n",
|
||||
"[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\beta$ by computing their\n",
|
||||
"variances, evaluate the Mean Squared error (MSE)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n",
|
||||
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"and the $R^2$ score function. If $\\tilde{\\hat{y}}_i$ is the predicted\n",
|
||||
"value of the $i-th$ sample and $y_i$ is the corresponding true value,\n",
|
||||
"then the score $R^2$ is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where we have defined the mean value of $\\hat{y}$ as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Your code has to include a scaling of the data (for example by\n",
|
||||
"subtracting the mean value), and\n",
|
||||
"a split of the data in training and test data. For this exercise you can\n",
|
||||
"either write your own code or use for example the function for\n",
|
||||
"splitting training data provided by the library **Scikit-Learn** (make\n",
|
||||
"sure you have installed it). This function is called\n",
|
||||
"$train\\_test\\_split$. **You should present a critical discussion of why and how you have scaled or not scaled the data**.\n",
|
||||
"\n",
|
||||
"It is normal in essentially all Machine Learning studies to split the\n",
|
||||
"data in a training set and a test set (eventually also an additional\n",
|
||||
"validation set). There\n",
|
||||
"is no explicit recipe for how much data should be included as training\n",
|
||||
"data and say test data. An accepted rule of thumb is to use\n",
|
||||
"approximately $2/3$ to $4/5$ of the data as training data.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"You can easily reuse the solutions to your exercises from week 35 and week 36.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Exercise: Bias-variance trade-off and resampling techniques\n",
|
||||
"\n",
|
||||
"Our aim here is to study the bias-variance trade-off by implementing the **bootstrap** resampling technique.\n",
|
||||
"\n",
|
||||
"With a code which does OLS and includes resampling techniques, \n",
|
||||
"we will now discuss the bias-variance trade-off in the context of\n",
|
||||
"continuous predictions such as regression. However, many of the\n",
|
||||
"intuitions and ideas discussed here also carry over to classification\n",
|
||||
"tasks and basically all Machine Learning algorithms. \n",
|
||||
"\n",
|
||||
"Before you perform an analysis of the bias-variance trade-off on your test data, make\n",
|
||||
"first a figure similar to Fig. 2.11 of Hastie, Tibshirani, and\n",
|
||||
"Friedman. Figure 2.11 of this reference displays only the test and training MSEs. The test MSE can be used to \n",
|
||||
"indicate possible regions of low/high bias and variance. You will most likely not get an\n",
|
||||
"equally smooth curve!\n",
|
||||
"\n",
|
||||
"With this result we move on to the bias-variance trade-off analysis.\n",
|
||||
"\n",
|
||||
"Consider a\n",
|
||||
"dataset $\\mathcal{L}$ consisting of the data\n",
|
||||
"$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n",
|
||||
"\n",
|
||||
"Let us assume that the true data is generated from a noisy model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{y}=f(\\boldsymbol{x}) + \\boldsymbol{\\epsilon}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here $\\epsilon$ is normally distributed with mean zero and standard\n",
|
||||
"deviation $\\sigma^2$.\n",
|
||||
"\n",
|
||||
"In our derivation of the ordinary least squares method we defined then\n",
|
||||
"an approximation to the function $f$ in terms of the parameters\n",
|
||||
"$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
|
||||
"that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n",
|
||||
"\n",
|
||||
"The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the means\n",
|
||||
"squared error via the so-called cost function"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here the expected value $\\mathbb{E}$ is the sample value. \n",
|
||||
"\n",
|
||||
"Show that you can rewrite this as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\frac{1}{n}\\sum_i(f_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\sigma^2.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Explain what the terms mean, which one is the bias and which one is\n",
|
||||
"the variance and discuss their interpretations.\n",
|
||||
"\n",
|
||||
"Perform then a bias-variance analysis of the Franke function by\n",
|
||||
"studying the MSE value as function of the complexity of your model.\n",
|
||||
"\n",
|
||||
"Discuss the bias and variance trade-off as function\n",
|
||||
"of your model complexity (the degree of the polynomial) and the number\n",
|
||||
"of data points, and possibly also your training and test data using the **bootstrap** resampling method.\n",
|
||||
"\n",
|
||||
"Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Exercise: Cross-validation as resampling techniques, adding more complexity\n",
|
||||
"\n",
|
||||
"The aim here is to write your own code for another widely popular\n",
|
||||
"resampling technique, the so-called cross-validation method. Again,\n",
|
||||
"before you start with cross-validation approach, you should scale your\n",
|
||||
"data.\n",
|
||||
"\n",
|
||||
"Implement the $k$-fold cross-validation algorithm (write your own\n",
|
||||
"code) and evaluate again the MSE function resulting\n",
|
||||
"from the test folds. You can compare your own code with that from\n",
|
||||
"**Scikit-Learn** if needed. \n",
|
||||
"\n",
|
||||
"Compare the MSE you get from your cross-validation code with the one\n",
|
||||
"you got from your **bootstrap** code. Comment your results. Try $5-10$\n",
|
||||
"folds. You can also compare your own cross-validation code with the\n",
|
||||
"one provided by **Scikit-Learn**.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Exercise: Ridge Regression on the Franke function with resampling\n",
|
||||
"\n",
|
||||
"Write your own code for the Ridge method, either using matrix\n",
|
||||
"inversion or the singular value decomposition as done in the previous\n",
|
||||
"exercise. Perform the same bootstrap analysis as in the\n",
|
||||
"Exercise 2 (for the same polynomials) and the cross-validation in exercise 3 but now for different values of $\\lambda$. Compare and\n",
|
||||
"analyze your results with those obtained in exercises 1-3. Study the\n",
|
||||
"dependence on $\\lambda$.\n",
|
||||
"\n",
|
||||
"Study also the bias-variance trade-off as function of various values of\n",
|
||||
"the parameter $\\lambda$. For the bias-variance trade-off, use the **bootstrap** resampling method. Comment your results. \n",
|
||||
"\n",
|
||||
"### Exercise: Lasso Regression on the Franke function with resampling\n",
|
||||
"\n",
|
||||
"This exercise is essentially a repeat of the previous two ones, but now\n",
|
||||
"with Lasso regression. Write either your own code (difficult and optional) or, in this case,\n",
|
||||
"you can also use the functionalities of **Scikit-Learn** (recommended). \n",
|
||||
"Give a\n",
|
||||
"critical discussion of the three methods and a judgement of which\n",
|
||||
"model fits the data best. Perform here as well an analysis of the bias-variance trade-off using the **bootstrap** resampling technique and an analysis of the mean squared error using cross-validation. \n",
|
||||
"\n",
|
||||
"### Exercise: Analysis of real data\n",
|
||||
"\n",
|
||||
"With our codes functioning and having been tested properly on a\n",
|
||||
"simpler function we are now ready to look at real data. We will\n",
|
||||
"essentially repeat in this exercise what was done in exercises 1-5. However, we\n",
|
||||
"need first to download the data and prepare properly the inputs to our\n",
|
||||
"codes. We are going to download digital terrain data from the website\n",
|
||||
"<https://earthexplorer.usgs.gov/>,\n",
|
||||
"\n",
|
||||
"Or, if you prefer, we have placed selected datafiles at <https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects/2021/Project1/DataFiles>\n",
|
||||
"\n",
|
||||
"In order to obtain data for a specific region, you need to register as\n",
|
||||
"a user (free) at this website and then decide upon which area you want\n",
|
||||
"to fetch the digital terrain data from. In order to be able to read\n",
|
||||
"the data properly, you need to specify that the format should be **SRTM\n",
|
||||
"Arc-Second Global** and download the data as a **GeoTIF** file. The\n",
|
||||
"files are then stored in *tif* format which can be imported into a\n",
|
||||
"Python program using"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"scipy.misc.imread"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here is a simple part of a Python code which reads and plots the data\n",
|
||||
"from such files"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"from imageio import imread\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from mpl_toolkits.mplot3d import Axes3D\n",
|
||||
"from matplotlib import cm\n",
|
||||
"\n",
|
||||
"# Load the terrain\n",
|
||||
"terrain1 = imread('SRTM_data_Norway_1.tif')\n",
|
||||
"# Show the terrain\n",
|
||||
"plt.figure()\n",
|
||||
"plt.title('Terrain over Norway 1')\n",
|
||||
"plt.imshow(terrain1, cmap='gray')\n",
|
||||
"plt.xlabel('X')\n",
|
||||
"plt.ylabel('Y')\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If you should have problems in downloading the digital terrain data,\n",
|
||||
"we provide two examples under the data folder of project 1. One is\n",
|
||||
"from a region close to Stavanger in Norway and the other Møsvatn\n",
|
||||
"Austfjell, again in Norway.\n",
|
||||
"Feel free to produce your own terrain data.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Alternatively, if you would like to use another data set, feel free to do so. This could be data close to your reseach area or simply a data set you found interesting. See for example [kaggle.com](https://www.kaggle.com/datasets) for examples.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Our final part deals with the parameterization of your digital terrain\n",
|
||||
"data (or your own data). We will apply all three methods for linear regression, the same type (or higher order) of polynomial\n",
|
||||
"approximation and cross-validation as resampling technique to evaluate which\n",
|
||||
"model fits the data best.\n",
|
||||
"\n",
|
||||
"At the end, you should present a critical evaluation of your results\n",
|
||||
"and discuss the applicability of these regression methods to the type\n",
|
||||
"of data presented here (either the terrain data we propose or other data sets)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {},
|
||||
|
||||
Reference in New Issue
Block a user