update
This commit is contained in:
Morten Hjorth-Jensen
@@ -588,154 +588,6 @@ which is our Lasso cost function!
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Test Function for what happens with OLS, Ridge and Lasso =====
|
||||
|
||||
We will play around with a study of the values for the optimal
|
||||
parameters $\bm{\beta}$ using OLS, Ridge and Lasso regression. For
|
||||
OLS, you will notice as function of the noise and polynomial degree,
|
||||
that the parameters $\beta$ will fluctuate from order to order in the
|
||||
polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS.
|
||||
|
||||
For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one.
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.model_selection import train_test_split
|
||||
from sklearn import linear_model
|
||||
|
||||
def R2(y_data, y_model):
|
||||
return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
|
||||
def MSE(y_data,y_model):
|
||||
n = np.size(y_model)
|
||||
return np.sum((y_data-y_model)**2)/n
|
||||
|
||||
# Make data set.
|
||||
n = 10000
|
||||
x = np.random.rand(n)
|
||||
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.randn(n)
|
||||
|
||||
Maxpolydegree = 5
|
||||
X = np.zeros((len(x),Maxpolydegree))
|
||||
X[:,0] = 1.0
|
||||
|
||||
for polydegree in range(1, Maxpolydegree+1):
|
||||
for degree in range(polydegree):
|
||||
X[:,degree] = x**(degree)
|
||||
|
||||
|
||||
# We split the data in test and training data
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
# matrix inversion to find beta
|
||||
OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
|
||||
print(OLSbeta)
|
||||
ypredictOLS = X_test @ OLSbeta
|
||||
print("Test MSE OLS")
|
||||
print(MSE(y_test,ypredictOLS))
|
||||
# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn
|
||||
# Decide which values of lambda to use
|
||||
nlambdas = 4
|
||||
MSERidgePredict = np.zeros(nlambdas)
|
||||
MSELassoPredict = np.zeros(nlambdas)
|
||||
lambdas = np.logspace(-3, 1, nlambdas)
|
||||
for i in range(nlambdas):
|
||||
lmb = lambdas[i]
|
||||
# Make the fit using Ridge and Lasso
|
||||
RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
|
||||
RegRidge.fit(X_train,y_train)
|
||||
RegLasso = linear_model.Lasso(lmb,fit_intercept=False)
|
||||
RegLasso.fit(X_train,y_train)
|
||||
# and then make the prediction
|
||||
ypredictRidge = RegRidge.predict(X_test)
|
||||
ypredictLasso = RegLasso.predict(X_test)
|
||||
# Compute the MSE and print it
|
||||
MSERidgePredict[i] = MSE(y_test,ypredictRidge)
|
||||
MSELassoPredict[i] = MSE(y_test,ypredictLasso)
|
||||
print(lmb,RegRidge.coef_)
|
||||
print(lmb,RegLasso.coef_)
|
||||
# Now plot the results
|
||||
plt.figure()
|
||||
plt.plot(np.log10(lambdas), MSERidgePredict, 'b', label = 'MSE Ridge Test')
|
||||
plt.plot(np.log10(lambdas), MSELassoPredict, 'r', label = 'MSE Lasso Test')
|
||||
plt.xlabel('log10(lambda)')
|
||||
plt.ylabel('MSE')
|
||||
plt.legend()
|
||||
plt.show()
|
||||
|
||||
|
||||
!ec
|
||||
|
||||
How can we understand this?
|
||||
|
||||
|
||||
!split
|
||||
===== Rerunning the above code =====
|
||||
|
||||
Let us write out the values of the coefficients $\beta_i$ as functions
|
||||
of the polynomial degree and noise. We will focus only on the Ridge
|
||||
results and some few selected values of the hyperparameter $\lambda$.
|
||||
|
||||
If we don't include any noise and run this code for different values
|
||||
of the polynomial degree, we notice that the results for $\beta_i$ do
|
||||
not show great changes from one order to the next. This is an
|
||||
indication that for higher polynomial orders, our parameters become
|
||||
less important.
|
||||
|
||||
If we however add noise, what happens is that the polynomial fit is
|
||||
trying to adjust the fit to traverse in the best possible way all data
|
||||
points. This can lead to large fluctuations in the parameters
|
||||
$\beta_i$ as functions of polynomial order. It will also be reflected
|
||||
in a larger value of the variance of each parameter $\beta_i$. What
|
||||
Ridge regression (and Lasso as well) are doing then is to try to
|
||||
quench the fluctuations in the parameters of $\beta_i$ which have a
|
||||
large variance (normally for higher orders in the polynomial).
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from IPython.display import display
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.model_selection import train_test_split
|
||||
from sklearn import linear_model
|
||||
|
||||
# Make data set.
|
||||
n = 1000
|
||||
x = np.random.rand(n)
|
||||
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.randn(n)
|
||||
|
||||
Maxpolydegree = 5
|
||||
X = np.zeros((len(x),Maxpolydegree))
|
||||
X[:,0] = 1.0
|
||||
|
||||
for polydegree in range(1, Maxpolydegree+1):
|
||||
for degree in range(polydegree):
|
||||
X[:,degree] = x**(degree)
|
||||
|
||||
|
||||
# We split the data in test and training data
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
# Decide which values of lambda to use
|
||||
nlambdas = 5
|
||||
lambdas = np.logspace(-3, 2, nlambdas)
|
||||
for i in range(nlambdas):
|
||||
lmb = lambdas[i]
|
||||
# Make the fit using Ridge only
|
||||
RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
|
||||
RegRidge.fit(X_train,y_train)
|
||||
# and then make the prediction
|
||||
ypredictRidge = RegRidge.predict(X_test)
|
||||
Coeffs = np.array(RegRidge.coef_)
|
||||
BetaValues = pd.DataFrame(Coeffs)
|
||||
BetaValues.columns = ['beta']
|
||||
display(BetaValues)
|
||||
|
||||
!ec
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
!split
|
||||
|
||||
Reference in New Issue
Block a user