Resampling approaches can be computationally expensive
+
+
+
+Resampling approaches can be computationally expensive, because they
+involve fitting the same statistical method multiple times using
+different subsets of the training data. However, due to recent
+advances in computing power, the computational requirements of
+resampling methods generally are not prohibitive. In this chapter, we
+discuss two of the most commonly used resampling methods,
+cross-validation and the bootstrap. Both methods are important tools
+in the practical application of many statistical learning
+procedures. For example, cross-validation can be used to estimate the
+test error associated with a given statistical learning method in
+order to evaluate its performance, or to select the appropriate level
+of flexibility. The process of evaluating a model’s performance is
+known as model assessment, whereas the process of selecting the proper
+level of flexibility for a model is known as model selection. The
+bootstrap is widely used.
+
+A popular strategy is to choose a penalty parameter that yields a good
+but parsimonious model. Information criteria measure the balance
+between model fit and model complexity. One possibility is Aikaike's
+information criterion (AIC).
+The AIC measures model fit by the log-likelihood
+and model complexity is measured by the number of parameters used by
+the model. The number of model parameters in regular regression simply
+corresponds to the number of covariates in the model. Or, by the
+degrees of freedom consumed by the model, which is equivalent to the
+trace of the hat matrix. For ridge regression it thus seems natural to
+define model complexity analogously by the trace of the ridge hat
+matrix. This yields the AIC for the linear regression model with ridge
+estimates:
+
+$$
+\begin{align*}
+\mbox{AIC}(\lambda) & = 2 \, p - 2 \log(\hat{L})
+\\
+& = 2 \, \mbox{tr} [\mathbf{H}(\lambda)] - 2 \log\{L[\hat{\beta}(\lambda), \hat{\sigma}^2(\lambda)]\}
+\\
+& = 2 \, \sum_{j=1}^p \frac{d_{jj}^2}{d_{jj}^2 + \lambda}
++ 2 n \, \log[\sqrt{2 \, \pi} \, \hat{\sigma}(\lambda)] + \frac{1}{\hat{\sigma}^2(\lambda)} \sum_{i=1}^n [y_i - \mathbf{X}_{i, \ast} \, \hat{\beta}(\lambda)]^2.
+\end{align*}
+$$
+
+The value of \( \lambda \) which minimizes \( \mbox{AIC}(\lambda) \) corresponds to the `optimal' balance of model complexity and overfitting.
+
+
+Instead of choosing the penalty parameter to balance model fit with
+model complexity, cross-validation requires it (i.e. the penalty
+parameter) to yield a model with good prediction
+performance. Commonly, this performance is evaluated on novel
+data. Novel data need not be easy to come by and one has to make do
+with the data at hand. The setting of `original' and novel data is
+then mimicked by sample splitting: the data set is divided into two
+(groups of samples). One of these two data sets, called the training
+set, plays the role of `original' data on which the model is
+built. The second of these data sets, called the test set, plays the
+role of the `novel' data and is used to evaluate the prediction
+performance (often operationalized as the log-likelihood or the
+prediction error or its square or the R2 score) of the model built on the training data set. This
+procedure (model building and prediction evaluation on training and
+test set, respectively) is done for a collection of possible penalty
+parameter choices. The penalty parameter that yields the model with
+the best prediction performance is to be preferred. The thus obtained
+performance evaluation depends on the actual split of the data set. To
+remove this dependence the data set is split many times into a
+training and test set. For each split the model parameters are
+estimated for all choices of \( \lambda \) using the training data and
+estimated parameters are evaluated on the corresponding test set. The
+penalty parameter that on average over the test sets performs best (in
+some sense) is then selected.
+
+
+The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks:
+
+
+
The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
+
In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.
+When the repetitive splitting of the data set is done randomly,
+samples may accidently end up in a fast majority of the splits in
+either training or test set. Such samples may have an unbalanced
+influence on either model building or prediction evaluation. To avoid
+this \( k \)-fold cross-validation structures the data splitting. The
+samples are divided into \( k \) more or less equally sized exhaustive and
+mutually exclusive subsets. In turn (at each split) one of these
+subsets plays the role of the test set while the union of the
+remaining subsets constitutes the training set. Such a splitting
+warrants a balanced representation of each sample in both training and
+test set over the splits. Still the division into the \( k \) subsets
+involves a degree of randomness. This may be fully excluded when
+choosing \( k=n \). This particular case is referred to as leave-one-out
+cross-validation (LOOCV).
+
+
How to set up the cross-validation for Ridge and/or Lasso
+
+
+
Define a range of interest for the penalty parameter.
+
Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
+
Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \hat{\sigma}_{-i}^2(\lambda) \), as
Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \hat{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
+
Repeat steps 1) to 3) such that each sample plays the role of the test set once.
+
Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
+Another approach in the LOOCV scheme is to the use the so-called Predicted Residual Error Sum of Squares (PRESS).
+
+
+We can define the optimal penalty parameter to minimize
+$$
+\begin{align*}
+\lambda_{\mbox{{\tiny opt}}} = \arg \min_{\lambda} \frac{1}{n} \sum_{i=1}^n [y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)]^2.
+\end{align*}
+$$
+
+
+The LOOCV prediction performance can be
+expressed analytically in terms of the known quantities derived from
+the design matrix and the parameters \( \beta \).
+
+Bootstrapping is a nonparametric approach to statistical inference
+that substitutes computation for more traditional distributional
+assumptions and asymptotic results. Bootstrapping offers a number of
+advantages:
+
+
+
The bootstrap is quite general, although there are some cases in which it fails.
+
Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.
+
It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
+
It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
Define a range of interest for the penalty parameter.
Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
-
Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set as
+
Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \hat{\sigma}_{-i}^2(\lambda) \), as
$$
@@ -1919,22 +1919,20 @@ $$
$$
-and the corresponding estimate of the error variance \( \hat{\sigma}_{-i}^2(\lambda) \).
Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \hat{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
Repeat steps 1) to 3) such that each sample plays the role of the test set once.
-
Average the prediction performances of the test sets at each grid point of the penalty bias/parameter
+
Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
-The quantity above is called the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.
The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
Define a range of interest for the penalty parameter.
Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
-
Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set as
+
Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \hat{\sigma}_{-i}^2(\lambda) \), as
$$
@@ -1877,21 +1877,19 @@ $$
\end{align*}
$$
-and the corresponding estimate of the error variance \( \hat{\sigma}_{-i}^2(\lambda) \).
Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \hat{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
Repeat steps 1) to 3) such that each sample plays the role of the test set once.
-
Average the prediction performances of the test sets at each grid point of the penalty bias/parameter
+
Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
$$
\begin{align*}
-\frac{1}{n} \sum_{i = 1}^n \log\{L[Y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}.
+\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}.
\end{align*}
$$
-The quantity above is called the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.
The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
Define a range of interest for the penalty parameter.
Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
-
Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set as
+
Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \hat{\sigma}_{-i}^2(\lambda) \), as
$$
@@ -1882,21 +1882,19 @@ $$
\end{align*}
$$
-and the corresponding estimate of the error variance \( \hat{\sigma}_{-i}^2(\lambda) \).
Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \hat{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
Repeat steps 1) to 3) such that each sample plays the role of the test set once.
-
Average the prediction performances of the test sets at each grid point of the penalty bias/parameter
+
Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
$$
\begin{align*}
-\frac{1}{n} \sum_{i = 1}^n \log\{L[Y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}.
+\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}.
\end{align*}
$$
-The quantity above is called the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.
The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
diff --git a/doc/pub/Regression/ipynb/Regression.ipynb b/doc/pub/Regression/ipynb/Regression.ipynb
index e36087719..09033bf6f 100644
--- a/doc/pub/Regression/ipynb/Regression.ipynb
+++ b/doc/pub/Regression/ipynb/Regression.ipynb
@@ -2271,7 +2271,7 @@
"\n",
"2. Divide the data set into training and test set comprising samples $\\{1, \\ldots, n\\} \\setminus i$ and $\\{ i \\}$, respectively.\n",
"\n",
- "3. Fit the linear regression model by means of ridge estimation for each $\\lambda$ in the grid using the training set as"
+ "3. Fit the linear regression model by means of ridge estimation for each $\\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\\hat{\\sigma}_{-i}^2(\\lambda)$, as"
]
},
{
@@ -2291,13 +2291,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "and the corresponding estimate of the error variance $\\hat{\\sigma}_{-i}^2(\\lambda)$.\n",
- "\n",
"1. Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\hat{X}_{i, \\ast}; \\hat{\\beta}_{-i}(\\lambda), \\hat{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\hat{X}_{i, \\ast} \\hat{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
"\n",
"2. Repeat steps 1) to 3) such that each sample plays the role of the test set once.\n",
"\n",
- "3. Average the prediction performances of the test sets at each grid point of the penalty bias/parameter"
+ "3. Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the *cross-validated log-likelihood*. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as"
]
},
{
@@ -2306,7 +2304,7 @@
"source": [
"$$\n",
"\\begin{align*}\n",
- "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[Y_i, \\mathbf{X}_{i, \\ast}; \\hat{\\beta}_{-i}(\\lambda), \\hat{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
+ "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\hat{\\beta}_{-i}(\\lambda), \\hat{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
"\\end{align*}\n",
"$$"
]
@@ -2315,8 +2313,6 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "The quantity above is called the *cross-validated log-likelihood*. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.\n",
- "\n",
"1. The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.\n",
"\n",
"## Predicted Residual Error Sum of Squares\n",
diff --git a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz
index 39e17ea5b..c0e43e801 100644
Binary files a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz and b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf
index 89eb8b91d..38f2d7194 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf and b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer.pdf b/doc/pub/Regression/pdf/Regression-beamer.pdf
index 3645fa90a..3b5a3ea7d 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer.pdf and b/doc/pub/Regression/pdf/Regression-beamer.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-minted.pdf b/doc/pub/Regression/pdf/Regression-minted.pdf
index 3ec432215..25211c937 100644
Binary files a/doc/pub/Regression/pdf/Regression-minted.pdf and b/doc/pub/Regression/pdf/Regression-minted.pdf differ
diff --git a/doc/src/Regression/Regression.do.txt b/doc/src/Regression/Regression.do.txt
index 67ab61560..605835410 100644
--- a/doc/src/Regression/Regression.do.txt
+++ b/doc/src/Regression/Regression.do.txt
@@ -1543,27 +1543,25 @@ o Define a range of interest for the penalty parameter.
o Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively.
-o Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set as
+o Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\hat{\sigma}_{-i}^2(\lambda)$, as
!bt
\begin{align*}
\hat{\beta}_{-i}(\lambda) & = ( \hat{X}_{-i, \ast}^{\top}
\hat{X}_{-i, \ast} + \lambda \hat{I}_{pp})^{-1}
\hat{X}_{-i, \ast}^{\top} \hat{y}_{-i}
\end{align*}
-!et
-and the corresponding estimate of the error variance $\hat{\sigma}_{-i}^2(\lambda)$.
+!et
o Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \hat{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function.
o Repeat steps 1) to 3) such that each sample plays the role of the test set once.
-o Average the prediction performances of the test sets at each grid point of the penalty bias/parameter
+o Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the *cross-validated log-likelihood*. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
!bt
\begin{align*}
-\frac{1}{n} \sum_{i = 1}^n \log\{L[Y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}.
+\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}.
\end{align*}
!et
-The quantity above is called the *cross-validated log-likelihood*. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.
o The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.