lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:00:16 +02:00
parent 1af325e7fb
commit 30d664a82b
9 changed files with 2392 additions and 942 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/chapter3.ipynb
+197 -190
View File
@@ -575,196 +575,6 @@
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Various steps in cross-validation\n",
"\n",
"When the repetitive splitting of the data set is done randomly,\n",
"samples may accidently end up in a fast majority of the splits in\n",
"either training or test set. Such samples may have an unbalanced\n",
"influence on either model building or prediction evaluation. To avoid\n",
"this $k$-fold cross-validation structures the data splitting. The\n",
"samples are divided into $k$ more or less equally sized exhaustive and\n",
"mutually exclusive subsets. In turn (at each split) one of these\n",
"subsets plays the role of the test set while the union of the\n",
"remaining subsets constitutes the training set. Such a splitting\n",
"warrants a balanced representation of each sample in both training and\n",
"test set over the splits. Still the division into the $k$ subsets\n",
"involves a degree of randomness. This may be fully excluded when\n",
"choosing $k=n$. This particular case is referred to as leave-one-out\n",
"cross-validation (LOOCV). \n",
"\n",
"\n",
"* Define a range of interest for the penalty parameter.\n",
"\n",
"* Divide the data set into training and test set comprising samples $\\{1, \\ldots, n\\} \\setminus i$ and $\\{ i \\}$, respectively.\n",
"\n",
"* Fit the linear regression model by means of ridge estimation for each $\\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\\boldsymbol{\\sigma}_{-i}^2(\\lambda)$, as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
"\\boldsymbol{\\beta}_{-i}(\\lambda) & = ( \\boldsymbol{X}_{-i, \\ast}^{T}\n",
"\\boldsymbol{X}_{-i, \\ast} + \\lambda \\boldsymbol{I}_{pp})^{-1}\n",
"\\boldsymbol{X}_{-i, \\ast}^{T} \\boldsymbol{y}_{-i}\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
"\n",
"* Repeat the first three steps such that each sample plays the role of the test set once.\n",
"\n",
"* Average the prediction performances of the test sets at each grid point of the penalty bias/parameter. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
"\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For the various values of $k$\n",
"\n",
"1. shuffle the dataset randomly.\n",
"\n",
"2. Split the dataset into $k$ groups.\n",
"\n",
"3. For each unique group:\n",
"\n",
"a. Decide which group to use as set for test data\n",
"\n",
"b. Take the remaining groups as a training data set\n",
"\n",
"c. Fit a model on the training set and evaluate it on the test set\n",
"\n",
"d. Retain the evaluation score and discard the model\n",
"\n",
"\n",
"5. Summarize the model using the sample of model evaluation scores\n",
"\n",
"The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import KFold\n",
"from sklearn.linear_model import Ridge\n",
"from sklearn.model_selection import cross_val_score\n",
"from sklearn.preprocessing import PolynomialFeatures\n",
"\n",
"# A seed just to ensure that the random numbers are the same for every run.\n",
"# Useful for eventual debugging.\n",
"np.random.seed(3155)\n",
"\n",
"# Generate the data.\n",
"nsamples = 100\n",
"x = np.random.randn(nsamples)\n",
"y = 3*x**2 + np.random.randn(nsamples)\n",
"\n",
"## Cross-validation on Ridge regression using KFold only\n",
"\n",
"# Decide degree on polynomial to fit\n",
"poly = PolynomialFeatures(degree = 6)\n",
"\n",
"# Decide which values of lambda to use\n",
"nlambdas = 500\n",
"lambdas = np.logspace(-3, 5, nlambdas)\n",
"\n",
"# Initialize a KFold instance\n",
"k = 5\n",
"kfold = KFold(n_splits = k)\n",
"\n",
"# Perform the cross-validation to estimate MSE\n",
"scores_KFold = np.zeros((nlambdas, k))\n",
"\n",
"i = 0\n",
"for lmb in lambdas:\n",
" ridge = Ridge(alpha = lmb)\n",
" j = 0\n",
" for train_inds, test_inds in kfold.split(x):\n",
" xtrain = x[train_inds]\n",
" ytrain = y[train_inds]\n",
"\n",
" xtest = x[test_inds]\n",
" ytest = y[test_inds]\n",
"\n",
" Xtrain = poly.fit_transform(xtrain[:, np.newaxis])\n",
" ridge.fit(Xtrain, ytrain[:, np.newaxis])\n",
"\n",
" Xtest = poly.fit_transform(xtest[:, np.newaxis])\n",
" ypred = ridge.predict(Xtest)\n",
"\n",
" scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)\n",
"\n",
" j += 1\n",
" i += 1\n",
"\n",
"\n",
"estimated_mse_KFold = np.mean(scores_KFold, axis = 1)\n",
"\n",
"## Cross-validation using cross_val_score from sklearn along with KFold\n",
"\n",
"# kfold is an instance initialized above as:\n",
"# kfold = KFold(n_splits = k)\n",
"\n",
"estimated_mse_sklearn = np.zeros(nlambdas)\n",
"i = 0\n",
"for lmb in lambdas:\n",
" ridge = Ridge(alpha = lmb)\n",
"\n",
" X = poly.fit_transform(x[:, np.newaxis])\n",
" estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)\n",
"\n",
" # cross_val_score return an array containing the estimated negative mse for every fold.\n",
" # we have to the the mean of every array in order to get an estimate of the mse of the model\n",
" estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)\n",
"\n",
" i += 1\n",
"\n",
"## Plot and compare the slightly different ways to perform cross-validation\n",
"\n",
"plt.figure()\n",
"\n",
"plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')\n",
"plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')\n",
"\n",
"plt.xlabel('log10(lambda)')\n",
"plt.ylabel('mse')\n",
"\n",
"plt.legend()\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -1221,6 +1031,203 @@
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cross-validation\n",
"\n",
"When the repetitive splitting of the data set is done randomly,\n",
"samples may accidently end up in a fast majority of the splits in\n",
"either training or test set. Such samples may have an unbalanced\n",
"influence on either model building or prediction evaluation. To avoid\n",
"this $k$-fold cross-validation structures the data splitting. The\n",
"samples are divided into $k$ more or less equally sized exhaustive and\n",
"mutually exclusive subsets. In turn (at each split) one of these\n",
"subsets plays the role of the test set while the union of the\n",
"remaining subsets constitutes the training set. Such a splitting\n",
"warrants a balanced representation of each sample in both training and\n",
"test set over the splits. Still the division into the $k$ subsets\n",
"involves a degree of randomness. This may be fully excluded when\n",
"choosing $k=n$. This particular case is referred to as leave-one-out\n",
"cross-validation (LOOCV). \n",
"\n",
"\n",
"* Define a range of interest for the penalty parameter.\n",
"\n",
"* Divide the data set into training and test set comprising samples $\\{1, \\ldots, n\\} \\setminus i$ and $\\{ i \\}$, respectively.\n",
"\n",
"* Fit the linear regression model by means of ridge estimation for each $\\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\\boldsymbol{\\sigma}_{-i}^2(\\lambda)$, as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
"\\boldsymbol{\\beta}_{-i}(\\lambda) & = ( \\boldsymbol{X}_{-i, \\ast}^{T}\n",
"\\boldsymbol{X}_{-i, \\ast} + \\lambda \\boldsymbol{I}_{pp})^{-1}\n",
"\\boldsymbol{X}_{-i, \\ast}^{T} \\boldsymbol{y}_{-i}\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
"\n",
"* Repeat the first three steps such that each sample plays the role of the test set once.\n",
"\n",
"* Average the prediction performances of the test sets at each grid point of the penalty bias/parameter. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
"\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For the various values of $k$\n",
"\n",
"1. shuffle the dataset randomly.\n",
"\n",
"2. Split the dataset into $k$ groups.\n",
"\n",
"3. For each unique group:\n",
"\n",
"a. Decide which group to use as set for test data\n",
"\n",
"b. Take the remaining groups as a training data set\n",
"\n",
"c. Fit a model on the training set and evaluate it on the test set\n",
"\n",
"d. Retain the evaluation score and discard the model\n",
"\n",
"\n",
"5. Summarize the model using the sample of model evaluation scores\n",
"\n",
"The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import KFold\n",
"from sklearn.linear_model import Ridge\n",
"from sklearn.model_selection import cross_val_score\n",
"from sklearn.preprocessing import PolynomialFeatures\n",
"\n",
"# A seed just to ensure that the random numbers are the same for every run.\n",
"# Useful for eventual debugging.\n",
"np.random.seed(3155)\n",
"\n",
"# Generate the data.\n",
"nsamples = 100\n",
"x = np.random.randn(nsamples)\n",
"y = 3*x**2 + np.random.randn(nsamples)\n",
"\n",
"## Cross-validation on Ridge regression using KFold only\n",
"\n",
"# Decide degree on polynomial to fit\n",
"poly = PolynomialFeatures(degree = 6)\n",
"\n",
"# Decide which values of lambda to use\n",
"nlambdas = 500\n",
"lambdas = np.logspace(-3, 5, nlambdas)\n",
"\n",
"# Initialize a KFold instance\n",
"k = 5\n",
"kfold = KFold(n_splits = k)\n",
"\n",
"# Perform the cross-validation to estimate MSE\n",
"scores_KFold = np.zeros((nlambdas, k))\n",
"\n",
"i = 0\n",
"for lmb in lambdas:\n",
" ridge = Ridge(alpha = lmb)\n",
" j = 0\n",
" for train_inds, test_inds in kfold.split(x):\n",
" xtrain = x[train_inds]\n",
" ytrain = y[train_inds]\n",
"\n",
" xtest = x[test_inds]\n",
" ytest = y[test_inds]\n",
"\n",
" Xtrain = poly.fit_transform(xtrain[:, np.newaxis])\n",
" ridge.fit(Xtrain, ytrain[:, np.newaxis])\n",
"\n",
" Xtest = poly.fit_transform(xtest[:, np.newaxis])\n",
" ypred = ridge.predict(Xtest)\n",
"\n",
" scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)\n",
"\n",
" j += 1\n",
" i += 1\n",
"\n",
"\n",
"estimated_mse_KFold = np.mean(scores_KFold, axis = 1)\n",
"\n",
"## Cross-validation using cross_val_score from sklearn along with KFold\n",
"\n",
"# kfold is an instance initialized above as:\n",
"# kfold = KFold(n_splits = k)\n",
"\n",
"estimated_mse_sklearn = np.zeros(nlambdas)\n",
"i = 0\n",
"for lmb in lambdas:\n",
" ridge = Ridge(alpha = lmb)\n",
"\n",
" X = poly.fit_transform(x[:, np.newaxis])\n",
" estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)\n",
"\n",
" # cross_val_score return an array containing the estimated negative mse for every fold.\n",
" # we have to the the mean of every array in order to get an estimate of the mse of the model\n",
" estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)\n",
"\n",
" i += 1\n",
"\n",
"## Plot and compare the slightly different ways to perform cross-validation\n",
"\n",
"plt.figure()\n",
"\n",
"plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')\n",
"plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')\n",
"\n",
"plt.xlabel('log10(lambda)')\n",
"plt.ylabel('mse')\n",
"\n",
"plt.legend()\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"More examples of the application of cross-validation follow here."
]
},
{
"cell_type": "code",
"execution_count": null,