lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 38

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-14 08:00:55 +02:00
parent 46977b1c1b
commit 7bcbcaa1f3
50 changed files with 400 additions and 3030 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week38/week38.do.txt
+7 -179
View File
@@ -10,7 +10,7 @@ DATE: September 15-19, 2025
!bblock Material for the lecture on Monday September 15
o Statistical interpretation of Ridge and Lasso regression
o Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff (this may partly be discussed during the exercise sessions as well.
o See video on ADAgrad, RMSprop and ADAM (material from last week not covered during lecture) at URL:"https://youtu.be/"
o The material we did not cover last week, that is on more advanced methods for updating the learning rate, are covered by its own video. We will briefly discuss these topics at the beginning of the lecture and during the lab sessions. See video on ADAgrad, RMSprop and ADAM (material from last week not covered during lecture) at URL:"https://youtu.be/"
# * "Video of Lecture":"https://youtu.be/omLmp_kkie0"
# * "Whiteboard notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesSeptember9.pdf"
!eblock
@@ -24,6 +24,7 @@ o Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation)
o "Video on bias-variance tradeoff":"https://www.youtube.com/watch?v=EuBBz3bI-aA"
o "Video on Bootstrapping":"https://www.youtube.com/watch?v=Xz0x-8-cgaQ"
o "Video on cross validation":"https://www.youtube.com/watch?v=fSytzGwwBVw"
For the lab session, the following video on cross validation (from 2024), could be helpful, see URL:"https://www.youtube.com/watch?v=T9jjWsmsd1o"
!eblock
@@ -1374,182 +1375,9 @@ plt.show()
!split
===== Material for the lab sessions =====
This week we will discuss during the first hour of each lab session
some technicalities related to the project and methods for updating
the learning like ADAgrad, RMSprop and ADAM. As teaching material, see
the jupyter-notebook from week 37 (September 12-16).
!split
===== Plans for the lab sessions =====
!bblock Material for the active learning sessions on Tuesday and Wednesday
* bias-variance tradeoff
* Resampling techniques, cross-validation examples included here, see also the lectures from last week on the bootstrap method
* Exercise for week 38 on the bias-variance tradeoff, see also the video from the lab session from week 37 at URL:"https://youtu.be/omLmp_kkie0"
!eblock
!split
===== Lab session: Material relevant for the first project =====
!split
===== Various steps in cross-validation =====
When the repetitive splitting of the data set is done randomly,
samples may accidently end up in a fast majority of the splits in
either training or test set. Such samples may have an unbalanced
influence on either model building or prediction evaluation. To avoid
this $k$-fold cross-validation structures the data splitting. The
samples are divided into $k$ more or less equally sized exhaustive and
mutually exclusive subsets. In turn (at each split) one of these
subsets plays the role of the test set while the union of the
remaining subsets constitutes the training set. Such a splitting
warrants a balanced representation of each sample in both training and
test set over the splits. Still the division into the $k$ subsets
involves a degree of randomness. This may be fully excluded when
choosing $k=n$. This particular case is referred to as leave-one-out
cross-validation (LOOCV).
!split
===== How to set up the cross-validation for Ridge and/or Lasso =====
* Define a range of interest for the penalty parameter.
* Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively.
* Fit the linear regression model by means of for example Ridge or Lasso regression for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\bm{\sigma}_{-i}^2(\lambda)$, as
!bt
\begin{align*}
\bm{\beta}_{-i}(\lambda) & = ( \bm{X}_{-i, \ast}^{T}
\bm{X}_{-i, \ast} + \lambda \bm{I}_{pp})^{-1}
\bm{X}_{-i, \ast}^{T} \bm{y}_{-i}
\end{align*}
!et
* Evaluate the prediction performance of these models on the test set by $C[y_i, \bm{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]$. Or, by the prediction error $|y_i - \bm{X}_{i, \ast} \bm{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function.
* Repeat the first three steps such that each sample plays the role of the test set once.
* Average the prediction performances of the test sets at each grid point of the penalty bias/parameter. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.
!split
===== Cross-validation in brief =====
For the various values of $k$
o shuffle the dataset randomly.
o Split the dataset into $k$ groups.
o For each unique group:
o Decide which group to use as set for test data
o Take the remaining groups as a training data set
o Fit a model on the training set and evaluate it on the test set
o Retain the evaluation score and discard the model
o Summarize the model using the sample of model evaluation scores
!split
===== Code Example for Cross-validation and $k$-fold Cross-validation =====
The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial.
!bc pycod
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import PolynomialFeatures
# A seed just to ensure that the random numbers are the same for every run.
# Useful for eventual debugging.
np.random.seed(3155)
# Generate the data.
nsamples = 100
x = np.random.randn(nsamples)
y = 3*x**2 + np.random.randn(nsamples)
## Cross-validation on Ridge regression using KFold only
# Decide degree on polynomial to fit
poly = PolynomialFeatures(degree = 6)
# Decide which values of lambda to use
nlambdas = 500
lambdas = np.logspace(-3, 5, nlambdas)
# Initialize a KFold instance
k = 5
kfold = KFold(n_splits = k)
# Perform the cross-validation to estimate MSE
scores_KFold = np.zeros((nlambdas, k))
i = 0
for lmb in lambdas:
ridge = Ridge(alpha = lmb)
j = 0
for train_inds, test_inds in kfold.split(x):
xtrain = x[train_inds]
ytrain = y[train_inds]
xtest = x[test_inds]
ytest = y[test_inds]
Xtrain = poly.fit_transform(xtrain[:, np.newaxis])
ridge.fit(Xtrain, ytrain[:, np.newaxis])
Xtest = poly.fit_transform(xtest[:, np.newaxis])
ypred = ridge.predict(Xtest)
scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)
j += 1
i += 1
estimated_mse_KFold = np.mean(scores_KFold, axis = 1)
## Cross-validation using cross_val_score from sklearn along with KFold
# kfold is an instance initialized above as:
# kfold = KFold(n_splits = k)
estimated_mse_sklearn = np.zeros(nlambdas)
i = 0
for lmb in lambdas:
ridge = Ridge(alpha = lmb)
X = poly.fit_transform(x[:, np.newaxis])
estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)
# cross_val_score return an array containing the estimated negative mse for every fold.
# we have to the the mean of every array in order to get an estimate of the mse of the model
estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)
i += 1
## Plot and compare the slightly different ways to perform cross-validation
plt.figure()
plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')
plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')
plt.xlabel('log10(lambda)')
plt.ylabel('mse')
plt.legend()
plt.show()
!ec
For the lab session, the following video on cross validation (from 2024), could be helpful, see URL:"https://www.youtube.com/watch?v=T9jjWsmsd1o"