FYS-STK4155/doc/pub/week38/ipynb/week38.ipynb

Data Analysis and Machine Learning: Logistic Regression

Morten Hjorth-Jensen, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

Date: Oct 26, 2021

Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license

Plans for week 38

• Thursday: Summary of regression methods and discussion of project 1. Start Logistic Regression

• Video of Lecture September 23

• Friday: Logistic Regression and Optimization methods

• Video of Lecture September 24

Thursday September 23

Ridge and LASSO Regression, reminder

The expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is

{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.

or we can state it as

{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,

where we have used the definition of a norm-2 vector, that is

\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.

By minimizing the above equation with respect to the parameters \boldsymbol{\beta} we could then obtain an analytical expression for the parameters \boldsymbol{\beta}. We can add a regularization parameter \lambda by defining a new cost function to be optimized, that is

{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2

which leads to the Ridge regression minimization problem where we require that \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t, where t is a finite number larger than zero. By defining

C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,

we have a new optimization equation

{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1

which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

Here we have defined the norm-1 as

\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert.

Various steps in cross-validation

When the repetitive splitting of the data set is done randomly, samples may accidently end up in a fast majority of the splits in either training or test set. Such samples may have an unbalanced influence on either model building or prediction evaluation. To avoid this $k$-fold cross-validation structures the data splitting. The samples are divided into k more or less equally sized exhaustive and mutually exclusive subsets. In turn (at each split) one of these subsets plays the role of the test set while the union of the remaining subsets constitutes the training set. Such a splitting warrants a balanced representation of each sample in both training and test set over the splits. Still the division into the k subsets involves a degree of randomness. This may be fully excluded when choosing k=n. This particular case is referred to as leave-one-out cross-validation (LOOCV).

How to set up the cross-validation for Ridge and/or Lasso

• Define a range of interest for the penalty parameter.

• Divide the data set into training and test set comprising samples \{1, \ldots, n\} \setminus i and \{ i \}, respectively.

• Fit the linear regression model by means of ridge estimation for each \lambda in the grid using the training set, and the corresponding estimate of the error variance \boldsymbol{\sigma}_{-i}^2(\lambda), as

\begin{align*}
\boldsymbol{\beta}_{-i}(\lambda) & =  ( \boldsymbol{X}_{-i, \ast}^{T}
\boldsymbol{X}_{-i, \ast} + \lambda \boldsymbol{I}_{pp})^{-1}
\boldsymbol{X}_{-i, \ast}^{T} \boldsymbol{y}_{-i}
\end{align*}

• Evaluate the prediction performance of these models on the test set by C[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]. Or, by the prediction error |y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\beta}_{-i}(\lambda)|, the relative error, the error squared or the R2 score function.

• Repeat the first three steps such that each sample plays the role of the test set once.

• Average the prediction performances of the test sets at each grid point of the penalty bias/parameter. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.

Cross-validation in brief

For the various values of k

1. shuffle the dataset randomly.

2. Split the dataset into k groups.

3. For each unique group:

a. Decide which group to use as set for test data

b. Take the remaining groups as a training data set

c. Fit a model on the training set and evaluate it on the test set

d. Retain the evaluation score and discard the model

5. Summarize the model using the sample of model evaluation scores

Code Example for Cross-validation and $k$-fold Cross-validation

The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial.

In [1]:

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import PolynomialFeatures

# A seed just to ensure that the random numbers are the same for every run.
# Useful for eventual debugging.
np.random.seed(3155)

# Generate the data.
nsamples = 100
x = np.random.randn(nsamples)
y = 3*x**2 + np.random.randn(nsamples)

## Cross-validation on Ridge regression using KFold only

# Decide degree on polynomial to fit
poly = PolynomialFeatures(degree = 6)

# Decide which values of lambda to use
nlambdas = 500
lambdas = np.logspace(-3, 5, nlambdas)

# Initialize a KFold instance
k = 5
kfold = KFold(n_splits = k)

# Perform the cross-validation to estimate MSE
scores_KFold = np.zeros((nlambdas, k))

i = 0
for lmb in lambdas:
    ridge = Ridge(alpha = lmb)
    j = 0
    for train_inds, test_inds in kfold.split(x):
        xtrain = x[train_inds]
        ytrain = y[train_inds]

        xtest = x[test_inds]
        ytest = y[test_inds]

        Xtrain = poly.fit_transform(xtrain[:, np.newaxis])
        ridge.fit(Xtrain, ytrain[:, np.newaxis])

        Xtest = poly.fit_transform(xtest[:, np.newaxis])
        ypred = ridge.predict(Xtest)

        scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)

        j += 1
    i += 1


estimated_mse_KFold = np.mean(scores_KFold, axis = 1)

## Cross-validation using cross_val_score from sklearn along with KFold

# kfold is an instance initialized above as:
# kfold = KFold(n_splits = k)

estimated_mse_sklearn = np.zeros(nlambdas)
i = 0
for lmb in lambdas:
    ridge = Ridge(alpha = lmb)

    X = poly.fit_transform(x[:, np.newaxis])
    estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)

    # cross_val_score return an array containing the estimated negative mse for every fold.
    # we have to the the mean of every array in order to get an estimate of the mse of the model
    estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)

    i += 1

## Plot and compare the slightly different ways to perform cross-validation

plt.figure()

plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')
plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')

plt.xlabel('log10(lambda)')
plt.ylabel('mse')

plt.legend()

plt.show()

To think about, first part

When you are comparing your own code with for example Scikit-Learn's library, there are some technicalities to keep in mind. The examples here demonstrate some of these aspects with potential pitfalls.

The discussion here focuses on the role of the intercept, how we can set up the design matrix, what scaling we should use and other topics which tend confuse us.

The intercept can be interpreted as the expected value of our target/output variables when all other predictors are set to zero. Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. Furthermore, in for example Ridge and Lasso regression, the default solutions from the library Scikit-Learn (when not shrinking \beta_0) for the unknown parameters \boldsymbol{\beta}, are derived under the assumption that both \boldsymbol{y} and \boldsymbol{X} are zero centered, that is we subtract the mean values.

More thinking

If our predictors represent different scales, then it is important to standardize the design matrix \boldsymbol{X} by subtracting the mean of each column from the corresponding column and dividing the column with its standard deviation. Most machine learning libraries do this as a default. This means that if you compare your code with the results from a given library, the results may differ.

The Standadscaler function in Scikit-Learn does this for us. For the data sets we have been studying in our various examples, the data are in many cases already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform a survey of your data, with a critical assessment of them in case you need to scale the data.

If you need to scale the data, not doing so will give an unfair penalization of the parameters since their magnitude depends on the scale of their corresponding predictor.

Suppose as an example that you you have an input variable given by the heights of different persons. Human height might be measured in inches or meters or kilometers. If measured in kilometers, a standard linear regression model with this predictor would probably give a much bigger coefficient term, than if measured in millimeters. This can clearly lead to problems in evaluating the cost/loss functions.

Still thinking

Keep in mind that when you transform your data set before training a model, the same transformation needs to be done on your eventual new data set before making a prediction. If we translate this into a Python code, it would could be implemented as follows

In [3]:

#Model training, we compute the mean value of y and X
y_train_mean = np.mean(y_train)
X_train_mean = np.mean(X_train,axis=0)
X_train = X_train - X_train_mean
y_train = y_train - y_train_mean

# The we fit our model with the training data
trained_model = some_model.fit(X_train,y_train)


#Model prediction, we need also to transform our data set used for the prediction.
X_test = X_test - X_train_mean #Use mean from training data
y_pred = trained_model(X_test)
y_pred = y_pred + y_train_mean

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_42372/1132315548.py in <module>
      1 #Model training, we compute the mean value of y and X
----> 2 y_train_mean = np.mean(y_train)
      3 X_train_mean = np.mean(X_train,axis=0)
      4 X_train = X_train - X_train_mean
      5 y_train = y_train - y_train_mean

NameError: name 'y_train' is not defined

What does centering (subtracting the mean values) mean mathematically?

Let us try to understand what this may imply mathematically when we subtract the mean values, also known as zero centering. For simplicity, we will focus on ordinary regression, as done in the above example.

The cost/loss function for regression is

C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.

Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.

What we have done is to single out the \beta_0 term in the definition of the mean squared error (MSE). The design matrix X does in this case not contain any intercept column. When we take the derivative with respect to \beta_0, we want the derivative to obey

\frac{\partial C}{\partial \beta_j} = 0,

for all j. For \beta_0 we have

\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).

Multiplying away the constant 2/n, we obtain

\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.

Further Manipulations

Let us special first to the case where we have only two parameters \beta_0 and \beta_1. Our result for \beta_0 simplifies then to

n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.

We obtain then

\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.

If we define

\mu_1=\frac{1}{n}\sum_{i=0}^{n-1} (X_{i1},

and if we define the mean value of the outputs as

\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,

we have

\beta_0 = \mu_y - \beta_1\mu_{1}.

In the general case, that is we have more parameters than \beta_0 and \beta_1, we have

\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.

Replacing y_i with y_i - y_i - \overline{\boldsymbol{y}} and centering also our design matrix results in a cost function (in vector-matrix disguise)

C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).

Wrapping it up

If we minimize with respect to \boldsymbol{\beta} we have then

\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},

where \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} and \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}.

For Ridge regression we need to add \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} to the cost function and get then

\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.

What does this mean? And why do we insist on all this? Let us look at some examples.

Linear Regression code, Intercept handling first

This code shows a simple first-order fit to a data set using the above transformed data, where we consider the role of the intercept first, by either excluding it or including it (code example thanks to Øyvind Sigmundson Schøyen). Here our scaling of the data is done by subtracting the mean values only. Note also that we do not split the data into training and test.

In [4]:

import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression


np.random.seed(2021)

def MSE(y_data,y_model):
    n = np.size(y_model)
    return np.sum((y_data-y_model)**2)/n


def fit_beta(X, y):
    return np.linalg.pinv(X.T @ X) @ X.T @ y


true_beta = [2, 0.5, 3.7]

x = np.linspace(0, 1, 11)
y = np.sum(
    np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0
) + 0.1 * np.random.normal(size=len(x))

degree = 3
X = np.zeros((len(x), degree))

# Include the intercept in the design matrix
for p in range(degree):
    X[:, p] = x ** p

beta = fit_beta(X, y)

# Intercept is included in the design matrix
skl = LinearRegression(fit_intercept=False).fit(X, y)

print(f"True beta: {true_beta}")
print(f"Fitted beta: {beta}")
print(f"Sklearn fitted beta: {skl.coef_}")
ypredictOwn = X @ beta
ypredictSKL = skl.predict(X)
print(f"MSE with intercept column")
print(MSE(y,ypredictOwn))
print(f"MSE with intercept column from SKL")
print(MSE(y,ypredictSKL))


plt.figure()
plt.scatter(x, y, label="Data")
plt.plot(x, X @ beta, label="Fit")
plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)")


# Do not include the intercept in the design matrix
X = np.zeros((len(x), degree - 1))

for p in range(degree - 1):
    X[:, p] = x ** (p + 1)

# Intercept is not included in the design matrix
skl = LinearRegression(fit_intercept=True).fit(X, y)

# Use centered values for X and y when computing coefficients
y_offset = np.average(y, axis=0)
X_offset = np.average(X, axis=0)

beta = fit_beta(X - X_offset, y - y_offset)
intercept = np.mean(y_offset - X_offset @ beta)

print(f"Manual intercept: {intercept}")
print(f"Fitted beta (wiothout intercept): {beta}")
print(f"Sklearn intercept: {skl.intercept_}")
print(f"Sklearn fitted beta (without intercept): {skl.coef_}")
ypredictOwn = X @ beta
ypredictSKL = skl.predict(X)
print(f"MSE with Manual intercept")
print(MSE(y,ypredictOwn+intercept))
print(f"MSE with Sklearn intercept")
print(MSE(y,ypredictSKL))

plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)")
plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)")
plt.grid()
plt.legend()

plt.show()

True beta: [2, 0.5, 3.7]
Fitted beta: [2.08376632 0.19569961 3.97898392]
Sklearn fitted beta: [2.08376632 0.19569961 3.97898392]
MSE with intercept column
0.004113634617443142
MSE with intercept column from SKL
0.004113634617443129
Manual intercept: 2.0837663229239056
Fitted beta (wiothout intercept): [0.19569961 3.97898392]
Sklearn intercept: 2.0837663229239043
Sklearn fitted beta (without intercept): [0.19569961 3.97898392]
MSE with Manual intercept
0.004113634617443132
MSE with Sklearn intercept
0.00411363461744314

The intercept is the value of our output/target variable when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case).

Printing the MSE, we see first that both methods give the same MSE, as they should. However, when we move to for example Ridge regression, the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the regularization term does not include \beta_0. For different values of \lambda, this may lead to differeing MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression is given by

\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,

but when we take out the intercept, this equation becomes

\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.

For Lasso regression we have

\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.

It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get a MSE which then contains the intercept.

Code Examples

Armed with this wisdom, we attempt first to simply set the intercept equal to False in our implementation of Ridge regression for our well-known vanilla data set.

In [5]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn import linear_model

def MSE(y_data,y_model):
    n = np.size(y_model)
    return np.sum((y_data-y_model)**2)/n


# A seed just to ensure that the random numbers are the same for every run.
# Useful for eventual debugging.
np.random.seed(3155)

n = 100
x = np.random.rand(n)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)

Maxpolydegree = 20
X = np.zeros((n,Maxpolydegree))
#We include explicitely the intercept column
for degree in range(Maxpolydegree):
    X[:,degree] = x**degree
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

p = Maxpolydegree
I = np.eye(p,p)
# Decide which values of lambda to use
nlambdas = 6
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
    lmb = lambdas[i]
    OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
    # Note: we include the intercept column and no scaling
    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
    RegRidge.fit(X_train,y_train)
    # and then make the prediction
    ytildeOwnRidge = X_train @ OwnRidgeBeta
    ypredictOwnRidge = X_test @ OwnRidgeBeta
    ytildeRidge = RegRidge.predict(X_train)
    ypredictRidge = RegRidge.predict(X_test)
    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
    print("Beta values for own Ridge implementation")
    print(OwnRidgeBeta)
    print("Beta values for Scikit-Learn Ridge implementation")
    print(RegRidge.coef_)
    print("MSE values for own Ridge implementation")
    print(MSEOwnRidgePredict[i])
    print("MSE values for Scikit-Learn Ridge implementation")
    print(MSERidgePredict[i])

# Now plot the results
plt.figure()
plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')
plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')

plt.xlabel('log10(lambda)')
plt.ylabel('MSE')
plt.legend()
plt.show()

Beta values for own Ridge implementation
[ 1.03032441e+00  6.28336218e-02 -6.24175744e-01  5.21169159e-02
  2.80847477e-01  2.12552073e-01  8.13220609e-02 -1.69634578e-02
 -6.50846112e-02 -7.38962192e-02 -5.94226022e-02 -3.50227564e-02
 -9.80609615e-03  1.08299273e-02  2.41882037e-02  2.93492130e-02
  2.64742912e-02  1.63249532e-02 -5.01831224e-05 -2.15098090e-02]
Beta values for Scikit-Learn Ridge implementation
[ 1.03032441e+00  6.28336218e-02 -6.24175744e-01  5.21169159e-02
  2.80847477e-01  2.12552073e-01  8.13220609e-02 -1.69634577e-02
 -6.50846112e-02 -7.38962192e-02 -5.94226022e-02 -3.50227564e-02
 -9.80609615e-03  1.08299273e-02  2.41882037e-02  2.93492130e-02
  2.64742912e-02  1.63249532e-02 -5.01831185e-05 -2.15098090e-02]
MSE values for own Ridge implementation
4.3632959128006053e-07
MSE values for Scikit-Learn Ridge implementation
4.363295916429748e-07
Beta values for own Ridge implementation
[ 1.03630548 -0.01963611 -0.37900111 -0.07062318  0.12182967  0.16343471
  0.13003291  0.07490892  0.02365049 -0.01449782 -0.03814292 -0.04909093
 -0.05009826 -0.04389027 -0.03279636 -0.01866537 -0.00289724  0.01348565
  0.02976145  0.04543942]
Beta values for Scikit-Learn Ridge implementation
[ 1.03630548 -0.01963611 -0.37900111 -0.07062318  0.12182967  0.16343471
  0.13003291  0.07490892  0.02365049 -0.01449782 -0.03814292 -0.04909093
 -0.05009826 -0.04389027 -0.03279636 -0.01866537 -0.00289724  0.01348565
  0.02976145  0.04543942]
MSE values for own Ridge implementation
5.1940428269262024e-06
MSE values for Scikit-Learn Ridge implementation
5.194042826836048e-06
Beta values for own Ridge implementation
[ 1.04220758 -0.10931453 -0.17641709 -0.06020587  0.02208512  0.05789007
  0.06491736  0.05785343  0.04537385  0.03196357  0.01969145  0.00934499
  0.00107405 -0.00526348 -0.00992331 -0.01318643 -0.01531845 -0.01655318
 -0.01708852 -0.01708781]
Beta values for Scikit-Learn Ridge implementation
[ 1.04220758 -0.10931453 -0.17641709 -0.06020587  0.02208512  0.05789007
  0.06491736  0.05785343  0.04537385  0.03196357  0.01969145  0.00934499
  0.00107405 -0.00526348 -0.00992331 -0.01318643 -0.01531845 -0.01655318
 -0.01708852 -0.01708781]
MSE values for own Ridge implementation
2.0940821989672366e-05
MSE values for Scikit-Learn Ridge implementation
2.0940821989628246e-05
Beta values for own Ridge implementation
[ 1.01219292 -0.06043581 -0.10391807 -0.05651951 -0.01898855  0.00312361
  0.01463049  0.01975848  0.02123176  0.02068067  0.01905883  0.01691985
  0.01458337  0.01223198  0.00996754  0.00784393  0.00588657  0.00410387
  0.00249435  0.00105081]
Beta values for Scikit-Learn Ridge implementation
[ 1.01219292 -0.06043581 -0.10391807 -0.05651951 -0.01898855  0.00312361
  0.01463049  0.01975848  0.02123176  0.02068067  0.01905883  0.01691985
  0.01458337  0.01223198  0.00996754  0.00784393  0.00588657  0.00410387
  0.00249435  0.00105081]
MSE values for own Ridge implementation
0.0003153514830958068
MSE values for Scikit-Learn Ridge implementation
0.00031535148309580843
Beta values for own Ridge implementation
[ 8.38916861e-01  1.31276579e-01  8.97497404e-03 -1.72271878e-02
 -2.11744554e-02 -1.91492986e-02 -1.57201944e-02 -1.23002365e-02
 -9.30466214e-03 -6.81048318e-03 -4.78184120e-03 -3.15130074e-03
 -1.84923989e-03 -8.13661243e-04  7.46984697e-06  6.56636616e-04
  1.16805821e-03  1.56912044e-03  1.88168312e-03  2.12318726e-03]
Beta values for Scikit-Learn Ridge implementation
[ 8.38916861e-01  1.31276579e-01  8.97497404e-03 -1.72271878e-02
 -2.11744554e-02 -1.91492986e-02 -1.57201944e-02 -1.23002365e-02
 -9.30466214e-03 -6.81048318e-03 -4.78184120e-03 -3.15130074e-03
 -1.84923989e-03 -8.13661243e-04  7.46984697e-06  6.56636616e-04
  1.16805821e-03  1.56912044e-03  1.88168312e-03  2.12318726e-03]
MSE values for own Ridge implementation
0.015072388895177166
MSE values for Scikit-Learn Ridge implementation
0.015072388895177053
Beta values for own Ridge implementation
[0.37396662 0.14174745 0.0764924  0.04892055 0.03447512 0.02586427
 0.02024962 0.01633913 0.01347916 0.0113104  0.0096208  0.00827728
 0.00719176 0.00630331 0.00556826 0.0049544  0.00443743 0.0039987
 0.0036237  0.003301  ]
Beta values for Scikit-Learn Ridge implementation
[0.37396662 0.14174745 0.0764924  0.04892055 0.03447512 0.02586427
 0.02024962 0.01633913 0.01347916 0.0113104  0.0096208  0.00827728
 0.00719176 0.00630331 0.00556826 0.0049544  0.00443743 0.0039987
 0.0036237  0.003301  ]
MSE values for own Ridge implementation
0.26409315307910025
MSE values for Scikit-Learn Ridge implementation
0.26409315307910025

The results here agree when we force Scikit-Learn's Ridge function to include the first column in our design matrix. We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. What happens if we do not include the intercept in our fit? Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).

Taking out the mean

In [6]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn import linear_model
from sklearn.preprocessing import StandardScaler

def MSE(y_data,y_model):
    n = np.size(y_model)
    return np.sum((y_data-y_model)**2)/n
# A seed just to ensure that the random numbers are the same for every run.
# Useful for eventual debugging.
np.random.seed(315)

n = 100
x = np.random.rand(n)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)

Maxpolydegree = 20
X = np.zeros((n,Maxpolydegree-1))

for degree in range(1,Maxpolydegree): #No intercept column
    X[:,degree-1] = x**(degree)

# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable
X_train_mean = np.mean(X_train,axis=0)
#Center by removing mean from each feature
X_train_scaled = X_train - X_train_mean 
X_test_scaled = X_test - X_train_mean
#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)
#Remove the intercept from the training data.
y_scaler = np.mean(y_train)           
y_train_scaled = y_train - y_scaler   

p = Maxpolydegree-1
I = np.eye(p,p)
# Decide which values of lambda to use
nlambdas = 6
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)

lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
    lmb = lambdas[i]
    OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
    intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data
    #Add intercept to prediction
    ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler 
    RegRidge = linear_model.Ridge(lmb)
    RegRidge.fit(X_train,y_train)
    ypredictRidge = RegRidge.predict(X_test)
    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
    print("Beta values for own Ridge implementation")
    print(OwnRidgeBeta) #Intercept is given by mean of target variable
    print("Beta values for Scikit-Learn Ridge implementation")
    print(RegRidge.coef_)
    print('Intercept from own implementation:')
    print(intercept_)
    print('Intercept from Scikit-Learn Ridge implementation')
    print(RegRidge.intercept_)
    print("MSE values for own Ridge implementation")
    print(MSEOwnRidgePredict[i])
    print("MSE values for Scikit-Learn Ridge implementation")
    print(MSERidgePredict[i])


# Now plot the results
plt.figure()
plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'b--', label = 'MSE own Ridge Test')
plt.plot(np.log10(lambdas), MSERidgePredict, 'g--', label = 'MSE SL Ridge Test')
plt.xlabel('log10(lambda)')
plt.ylabel('MSE')
plt.legend()
plt.show()

Beta values for own Ridge implementation
[ 3.43579948e-02 -5.43330971e-01 -3.10141414e-03  2.47116868e-01
  2.18613217e-01  1.02054837e-01 -4.25617659e-04 -5.90475506e-02
 -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02
  1.11482289e-02  2.88529063e-02  3.67047975e-02  3.38135733e-02
  2.02198702e-02 -3.46383925e-03 -3.63025821e-02]
Beta values for Scikit-Learn Ridge implementation
[ 3.43579948e-02 -5.43330971e-01 -3.10141413e-03  2.47116868e-01
  2.18613217e-01  1.02054837e-01 -4.25617657e-04 -5.90475506e-02
 -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02
  1.11482289e-02  2.88529063e-02  3.67047975e-02  3.38135733e-02
  2.02198702e-02 -3.46383925e-03 -3.63025821e-02]
Intercept from own implementation:
1.033030804518907
Intercept from Scikit-Learn Ridge implementation
1.03303080451838
MSE values for own Ridge implementation
3.139255959141931e-06
MSE values for Scikit-Learn Ridge implementation
3.1392559585690806e-06
Beta values for own Ridge implementation
[-0.05807125 -0.29822833 -0.08551306  0.08156108  0.13679863  0.12333649
  0.08251519  0.03815288  0.00111756 -0.02498832 -0.04010697 -0.04566964
 -0.04355837 -0.03562355 -0.02348765 -0.00848904  0.00831018  0.0260906
  0.04423486]
Beta values for Scikit-Learn Ridge implementation
[-0.05807125 -0.29822833 -0.08551306  0.08156108  0.13679863  0.12333649
  0.08251519  0.03815288  0.00111756 -0.02498832 -0.04010697 -0.04566964
 -0.04355837 -0.03562355 -0.02348765 -0.00848904  0.00831018  0.0260906
  0.04423486]
Intercept from own implementation:
1.0411487294305777
Intercept from Scikit-Learn Ridge implementation
1.0411487294305242
MSE values for own Ridge implementation
1.9601304850224855e-05
MSE values for Scikit-Learn Ridge implementation
1.9601304850078175e-05
Beta values for own Ridge implementation
[-0.1416398  -0.14021063 -0.05383795  0.01367553  0.04784395  0.05796251
  0.05447415  0.044613    0.03267527  0.02098261  0.01066519  0.00217499
 -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081  -0.01416528
 -0.01290947]
Beta values for Scikit-Learn Ridge implementation
[-0.1416398  -0.14021063 -0.05383795  0.01367553  0.04784395  0.05796251
  0.05447415  0.044613    0.03267527  0.02098261  0.01066519  0.00217499
 -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081  -0.01416528
 -0.01290947]
Intercept from own implementation:
1.04955699662783
Intercept from Scikit-Learn Ridge implementation
1.0495569966278266
MSE values for own Ridge implementation
5.495916150937975e-05
MSE values for Scikit-Learn Ridge implementation
5.495916150936546e-05
Beta values for own Ridge implementation
[-0.13535942 -0.08593216 -0.03568439 -0.0036367   0.01397146  0.02229529
  0.02503753  0.0245528   0.02228115  0.01908936  0.01549377  0.01179792
  0.00817631  0.00472512  0.00149311 -0.00149956 -0.00424967 -0.00676387
 -0.00905423]
Beta values for Scikit-Learn Ridge implementation
[-0.13535942 -0.08593216 -0.03568439 -0.0036367   0.01397146  0.02229529
  0.02503753  0.0245528   0.02228115  0.01908936  0.01549377  0.01179792
  0.00817631  0.00472512  0.00149311 -0.00149956 -0.00424967 -0.00676387
 -0.00905423]
Intercept from own implementation:
1.0399676689527961
Intercept from Scikit-Learn Ridge implementation
1.0399676689527975
MSE values for own Ridge implementation
7.571105947979378e-05
MSE values for Scikit-Learn Ridge implementation
7.571105947979412e-05
Beta values for own Ridge implementation
[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706  -0.00517114
 -0.00174276  0.00068734  0.00243186  0.00369758  0.00462287  0.0053018
  0.00579953  0.006162    0.00642221  0.00660427  0.00672607  0.0068011
  0.00683964]
Beta values for Scikit-Learn Ridge implementation
[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706  -0.00517114
 -0.00174276  0.00068734  0.00243186  0.00369758  0.00462287  0.0053018
  0.00579953  0.006162    0.00642221  0.00660427  0.00672607  0.0068011
  0.00683964]
Intercept from own implementation:
0.999955585168597
Intercept from Scikit-Learn Ridge implementation
0.999955585168597
MSE values for own Ridge implementation
0.0007698473260556331
MSE values for Scikit-Learn Ridge implementation
0.0007698473260556325
Beta values for own Ridge implementation
[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335
 -0.00323332 -0.00274989 -0.0023548  -0.00202756 -0.00175331 -0.00152117
 -0.001323   -0.0011526  -0.00100519 -0.00087697 -0.00076495 -0.00066668
 -0.00058016]
Beta values for Scikit-Learn Ridge implementation
[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335
 -0.00323332 -0.00274989 -0.0023548  -0.00202756 -0.00175331 -0.00152117
 -0.001323   -0.0011526  -0.00100519 -0.00087697 -0.00076495 -0.00066668
 -0.00058016]
Intercept from own implementation:
0.9637117593816477
Intercept from Scikit-Learn Ridge implementation
0.9637117593816477
MSE values for own Ridge implementation
0.0023813163025848865
MSE values for Scikit-Learn Ridge implementation
0.002381316302584886

We see here, when compared to the code which includes explicitely the intercept column, that our MSE value is actually smaller. This is because the regularization term does not include the intercept value \beta_0 in the fitting. This applies to Lasso regularization as well. It means that our optimization is now done only with the centered matrix and/or vector that enter the fitting procedure. Note also that the problem with the intercept occurs mainly in these type of polynomial fitting problem.

The next example is indeed an example where all these discussions about the role of intercept are not present.

More complicated Example: The Ising model

The one-dimensional Ising model with nearest neighbor interaction, no external field and a constant coupling constant J is given by

\begin{equation}
    H = -J \sum_{k}^L s_k s_{k + 1},
\label{_auto1} \tag{1}
\end{equation}

where s_i \in \{-1, 1\} and s_{N + 1} = s_1. The number of spins in the system is determined by L. For the one-dimensional system there is no phase transition.

We will look at a system of L = 40 spins with a coupling constant of J = 1. To get enough training data we will generate 10000 states with their respective energies.

In [ ]:

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable
import seaborn as sns
import scipy.linalg as scl
from sklearn.model_selection import train_test_split
import tqdm
sns.set(color_codes=True)
cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')

L = 40
n = int(1e4)

spins = np.random.choice([-1, 1], size=(n, L))
J = 1.0

energies = np.zeros(n)

for i in range(n):
    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))

Here we use ordinary least squares regression to predict the energy for the nearest neighbor one-dimensional Ising model on a ring, i.e., the endpoints wrap around. We will use linear regression to fit a value for the coupling constant to achieve this.

Reformulating the problem to suit regression

A more general form for the one-dimensional Ising model is

\begin{equation}
    H = - \sum_j^L \sum_k^L s_j s_k J_{jk}.
\label{_auto2} \tag{2}
\end{equation}

Here we allow for interactions beyond the nearest neighbors and a state dependent coupling constant. This latter expression can be formulated as a matrix-product

\begin{equation}
    \boldsymbol{H} = \boldsymbol{X} J,
\label{_auto3} \tag{3}
\end{equation}

where X_{jk} = s_j s_k and J is a matrix which consists of the elements -J_{jk}. This form of writing the energy fits perfectly with the form utilized in linear regression, that is

\begin{equation}
    \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon},
\label{_auto4} \tag{4}
\end{equation}

We split the data in training and test data as discussed in the previous example

In [ ]:

X = np.zeros((n, L ** 2))
for i in range(n):
    X[i] = np.outer(spins[i], spins[i]).ravel()
y = energies
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

Linear regression

In the ordinary least squares method we choose the cost function

\begin{equation}
    C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}.
\label{_auto5} \tag{5}
\end{equation}

We then find the extremal point of C by taking the derivative with respect to \boldsymbol{\beta} as discussed above. This yields the expression for \boldsymbol{\beta} to be

\boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}},

which immediately imposes some requirements on \boldsymbol{X} as there must exist an inverse of \boldsymbol{X}^T \boldsymbol{X}. If the expression we are modeling contains an intercept, i.e., a constant term, we must make sure that the first column of \boldsymbol{X} consists of 1. We do this here

In [ ]:

X_train_own = np.concatenate(
    (np.ones(len(X_train))[:, np.newaxis], X_train),
    axis=1
)
X_test_own = np.concatenate(
    (np.ones(len(X_test))[:, np.newaxis], X_test),
    axis=1
)

In [ ]:

def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:
    return scl.inv(x.T @ x) @ (x.T @ y)
beta = ols_inv(X_train_own, y_train)

Singular Value decomposition

Doing the inversion directly turns out to be a bad idea since the matrix \boldsymbol{X}^T\boldsymbol{X} is singular. An alternative approach is to use the singular value decomposition. Using the definition of the Moore-Penrose pseudoinverse we can write the equation for \boldsymbol{\beta} as

\boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y},

where the pseudoinverse of \boldsymbol{X} is given by

\boldsymbol{X}^{+} = \frac{\boldsymbol{X}^T}{\boldsymbol{X}^T\boldsymbol{X}}.

Using singular value decomposition we can decompose the matrix \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma} \boldsymbol{V}^T, where \boldsymbol{U} and \boldsymbol{V} are orthogonal(unitary) matrices and \boldsymbol{\Sigma} contains the singular values (more details below). where X^{+} = V\Sigma^{+} U^T. This reduces the equation for \omega to

\begin{equation}
    \boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}.
\label{_auto6} \tag{6}
\end{equation}

Note that solving this equation by actually doing the pseudoinverse (which is what we will do) is not a good idea as this operation scales as \mathcal{O}(n^3), where n is the number of elements in a general matrix. Instead, doing $QR$-factorization and solving the linear system as an equation would reduce this down to \mathcal{O}(n^2) operations.

In [ ]:

def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray:
    u, s, v = scl.svd(x)
    return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y

In [ ]:

beta = ols_svd(X_train_own,y_train)

When extracting the $J$-matrix we need to make sure that we remove the intercept, as is done here

In [ ]:

J = beta[1:].reshape(L, L)

A way of looking at the coefficients in J is to plot the matrices as images.

In [ ]:

fig = plt.figure(figsize=(20, 14))
im = plt.imshow(J, **cmap_args)
plt.title("OLS", fontsize=18)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
cb = fig.colorbar(im)
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
plt.show()

It is interesting to note that OLS considers both J_{j, j + 1} = -0.5 and J_{j, j - 1} = -0.5 as valid matrix elements for J. In our discussion below on hyperparameters and Ridge and Lasso regression we will see that this problem can be removed, partly and only with Lasso regression.

In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD?

The one-dimensional Ising model

Let us bring back the Ising model again, but now with an additional focus on Ridge and Lasso regression as well. We repeat some of the basic parts of the Ising model and the setup of the training and test data. The one-dimensional Ising model with nearest neighbor interaction, no external field and a constant coupling constant J is given by

\begin{equation}
    H = -J \sum_{k}^L s_k s_{k + 1},
\label{_auto7} \tag{7}
\end{equation}

where s_i \in \{-1, 1\} and s_{N + 1} = s_1. The number of spins in the system is determined by L. For the one-dimensional system there is no phase transition.

We will look at a system of L = 40 spins with a coupling constant of J = 1. To get enough training data we will generate 10000 states with their respective energies.

In [ ]:

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable
import seaborn as sns
import scipy.linalg as scl
from sklearn.model_selection import train_test_split
import sklearn.linear_model as skl
import tqdm
sns.set(color_codes=True)
cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')

L = 40
n = int(1e4)

spins = np.random.choice([-1, 1], size=(n, L))
J = 1.0

energies = np.zeros(n)

for i in range(n):
    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))

A more general form for the one-dimensional Ising model is

\begin{equation}
    H = - \sum_j^L \sum_k^L s_j s_k J_{jk}.
\label{_auto8} \tag{8}
\end{equation}

Here we allow for interactions beyond the nearest neighbors and a more adaptive coupling matrix. This latter expression can be formulated as a matrix-product on the form

\begin{equation}
    H = X J,
\label{_auto9} \tag{9}
\end{equation}

where X_{jk} = s_j s_k and J is the matrix consisting of the elements -J_{jk}. This form of writing the energy fits perfectly with the form utilized in linear regression, viz.

\begin{equation}
    \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}.
\label{_auto10} \tag{10}
\end{equation}

Warning:

Output truncated. This notebook contains too many cells to display efficiently.