where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) is the outcome of a +output (response, target, dependent or outcome) variable \( y_i \) is the outcome of a numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we -will treat \( y_i \) as our exact value for the response variable. +will treat \( y_i \) as our exact value for the output variable.
In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
diff --git a/doc/pub/week35/html/._week35-bs006.html b/doc/pub/week35/html/._week35-bs006.html index b02176fd6..ae33a7f1f 100644 --- a/doc/pub/week35/html/._week35-bs006.html +++ b/doc/pub/week35/html/._week35-bs006.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
+
# matrix inversion to find beta
+ # matrix inversion to find theta
# First we set up the data
import numpy as np
x = np.random.rand(100)
@@ -344,9 +344,9 @@ X[:,1] =
X[:,2] = x**2
X[:,3] = x**3
X[:,4] = x**4
-beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
+theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
# and then make the prediction
-ytilde = X @ beta
+ytilde = X @ theta
As we will also see in the first project, this may however not the be case in general and a standard matrix inversion -algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. +algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.
There is however a way to circumvent this problem and also diff --git a/doc/pub/week35/html/._week35-bs035.html b/doc/pub/week35/html/._week35-bs035.html index c3387ebb0..1b31c695a 100644 --- a/doc/pub/week35/html/._week35-bs035.html +++ b/doc/pub/week35/html/._week35-bs035.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is -a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) -we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). -The matrix has then a set of eigenpairs +
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. +$$ + +or we can state it as
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is
+$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +By minimizing the above equation with respect to the parameters +\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is
$$ -(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2 $$ -and the eigenvalues are given by the diagonal matrix
-$$ -\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). -$$ - -The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \)
-$$ -\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, -$$ - -with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \).
- -Not all square matrices are diagonalizable. A matrix like the one discussed above
-$$ -\boldsymbol{X} = \begin{bmatrix} -1& -1 \\ -1& -1\\ -\end{bmatrix} -$$ - -is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition -\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +
which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining
+$$ +C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1, +$$ + +we have a new optimization equation
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
+ +Here we have defined the norm-1 as
+$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + +
-
However, and this is the strength of the SVD algorithm, any general -matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and -two orthogonal/unitary matrices. The Singular Value Decompostion -(SVD) theorem -states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in -terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( m\times n \) -and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has -dimensionality \( m \times m \) and the last dimensionality \( n\times n \). -We have then -
- -$$ -\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T -$$ - -As an example, the above defective matrix can be decomposed as
+Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
$$ -\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, $$ -with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). -The SVD exits always! +
and +taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain +the optimal parameters +
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
+ +$$ +\sum_{i=0}^{p-1} \theta_i^2 \leq t, +$$ + +with \( t \) a finite positive number.
+ +If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
+ +When we compare this with the ordinary least squares result we have
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
+ +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
-The SVD -decomposition (singular values) gives eigenvalues -\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=p \), the -eigenvalues (singular values) are zero. +
When we have discussed the singular value decomposition of the design +matrix \( \boldsymbol{X} \), we will in turn perform a more rigorous mathematical +discussion of Ridge regression.
-In the general case, where our design matrix \( \boldsymbol{X} \) has dimension -\( n\times p \), the matrix is thus decomposed into an \( n\times n \) -orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) -and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) -singular values \( \sigma_i\geq 0 \) on the main diagonal and zeros filling -the rest of the matrix. There are at most \( p \) singular values -assuming that \( n > p \). In our regression examples for the nuclear -masses and the equation of state this is indeed the case, while for -the Ising model we have \( p > n \). These are often cases that lead to -near singular or singular matrices. -
+The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn.
-The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors.
+ + +import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
+
+def MSE(y_data,y_model):
+ n = np.size(y_model)
+ return np.sum((y_data-y_model)**2)/n
+
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+n = 100
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
+
+Maxpolydegree = 20
+X = np.zeros((n,Maxpolydegree))
+#We include explicitely the intercept column
+for degree in range(Maxpolydegree):
+ X[:,degree] = x**degree
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+p = Maxpolydegree
+I = np.eye(p,p)
+# Decide which values of lambda to use
+nlambdas = 6
+MSEOwnRidgePredict = np.zeros(nlambdas)
+MSERidgePredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 2, nlambdas)
+for i in range(nlambdas):
+ lmb = lambdas[i]
+ OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+ # Note: we include the intercept column and no scaling
+ RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+ RegRidge.fit(X_train,y_train)
+ # and then make the prediction
+ ytildeOwnRidge = X_train @ OwnRidgeTheta
+ ypredictOwnRidge = X_test @ OwnRidgeTheta
+ ytildeRidge = RegRidge.predict(X_train)
+ ypredictRidge = RegRidge.predict(X_test)
+ MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
+ MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+ print("Theta values for own Ridge implementation")
+ print(OwnRidgeTheta)
+ print("Theta values for Scikit-Learn Ridge implementation")
+ print(RegRidge.coef_)
+ print("MSE values for own Ridge implementation")
+ print(MSEOwnRidgePredict[i])
+ print("MSE values for Scikit-Learn Ridge implementation")
+ print(MSERidgePredict[i])
+
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')
+plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
+The results here agree when we force Scikit-Learn's Ridge function to include the first column in our design matrix. +We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. +What happens if we do not include the intercept in our fit? We will discuss this in more detail next week. +
diff --git a/doc/pub/week35/html/._week35-bs039.html b/doc/pub/week35/html/._week35-bs039.html index a285e77ef..802ba9af8 100644 --- a/doc/pub/week35/html/._week35-bs039.html +++ b/doc/pub/week35/html/._week35-bs039.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n -\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however -irrelevant in our calculations since they are multiplied with the -zeros in \( \boldsymbol{\Sigma} \). +
From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only if it is +a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) +we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). +The matrix has then a set of eigenpairs
-The economy-size decomposition removes extra rows or columns of zeros -from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns -in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. -Removing these zeros and columns can improve execution time -and reduce storage requirements without compromising the accuracy of -the decomposition. -
+$$ +(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), +$$ -If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). -If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). -The \( n=p \) case is obvious, we retain the full SVD. -In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. +
and the eigenvalues are given by the diagonal matrix
+$$ +\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +$$ + +The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \)
+$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \).
+ +Not all square matrices are diagonalizable. A matrix like the one discussed above
+$$ +\boldsymbol{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +$$ + +is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled.
diff --git a/doc/pub/week35/html/._week35-bs040.html b/doc/pub/week35/html/._week35-bs040.html index fa59903d9..039c223a7 100644 --- a/doc/pub/week35/html/._week35-bs040.html +++ b/doc/pub/week35/html/._week35-bs040.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
import numpy as np
-# SVD inversion
-def SVD(A):
- ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
- SVD is numerically more stable than the inversion algorithms provided by
- numpy and scipy.linalg at the cost of being slower.
- '''
- U, S, VT = np.linalg.svd(A,full_matrices=True)
- print('test U')
- print( (np.transpose(U) @ U - U @np.transpose(U)))
- print('test VT')
- print( (np.transpose(VT) @ VT - VT @np.transpose(VT)))
- print(U)
- print(S)
- print(VT)
-
- D = np.zeros((len(U),len(VT)))
- for i in range(0,len(VT)):
- D[i,i]=S[i]
- return U @ D @ VT
-
-
-X = np.array([ [1.0,-1.0], [1.0,-1.0]])
-#X = np.array([[1, 2], [3, 4], [5, 6]])
-
-print(X)
-C = SVD(X)
-# Print the difference between the original matrix and the SVD one
-print(C-X)
-
-The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first -column is the row-wise sum of the other two columns. The rank of a -matrix (the column rank) is the dimension of space spanned by the -column vectors. The rank of the matrix is the number of linearly -independent columns, in this case just \( 2 \). We see this from the -singular values when running the above code. Running the standard -inversion algorithm for matrix inversion with \( \boldsymbol{X}^T\boldsymbol{X} \) results -in the program terminating due to a singular matrix. +
However, and this is the strength of the SVD algorithm, any general +matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The Singular Value Decompostion +(SVD) theorem +states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in +terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( m\times n \) +and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has +dimensionality \( m \times m \) and the last dimensionality \( n\times n \). +We have then
+$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T +$$ + +As an example, the above defective matrix can be decomposed as
+ +$$ +\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! +
+ +The SVD +decomposition (singular values) gives eigenvalues +\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=p \), the +eigenvalues (singular values) are zero. +
+ +In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +\( n\times p \), the matrix is thus decomposed into an \( n\times n \) +orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) +and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) +singular values \( \sigma_i\geq 0 \) on the main diagonal and zeros filling +the rest of the matrix. There are at most \( p \) singular values +assuming that \( n > p \). In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have \( p > n \). These are often cases that lead to +near singular or singular matrices. +
+ +The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors.
+
-
The \( U \), \( S \), and \( V \) matrices returned from the svd() function -cannot be multiplied directly. +
If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however +irrelevant in our calculations since they are multiplied with the +zeros in \( \boldsymbol{\Sigma} \).
-As you can see from the code, the \( S \) vector must be converted into a -diagonal matrix. This may cause a problem as the size of the matrices -do not fit the rules of matrix multiplication, where the number of -columns in a matrix must match the number of rows in the subsequent -matrix. +
The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns +in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition.
-If you wish to include the zero singular values, you will need to -resize the matrices and set up a diagonal matrix as done in the above -example +
If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). +The \( n=p \) case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy.
diff --git a/doc/pub/week35/html/._week35-bs042.html b/doc/pub/week35/html/._week35-bs042.html index 7b7f72f0b..fe08b9cc9 100644 --- a/doc/pub/week35/html/._week35-bs042.html +++ b/doc/pub/week35/html/._week35-bs042.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
-Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
-$$ -\boldsymbol{X}=\begin{bmatrix} -x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ -x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ -x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ -\dots & \dots & \dots & \dots \dots & \dots \\ -x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ -x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ -\end{bmatrix}. -$$ + +import numpy as np
+# SVD inversion
+def SVD(A):
+ ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
+ SVD is numerically more stable than the inversion algorithms provided by
+ numpy and scipy.linalg at the cost of being slower.
+ '''
+ U, S, VT = np.linalg.svd(A,full_matrices=True)
+ print('test U')
+ print( (np.transpose(U) @ U - U @np.transpose(U)))
+ print('test VT')
+ print( (np.transpose(VT) @ VT - VT @np.transpose(VT)))
+ print(U)
+ print(S)
+ print(VT)
-We can SVD decompose our matrix as
-$$
-\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
-$$
+ D = np.zeros((len(U),len(VT)))
+ for i in range(0,len(VT)):
+ D[i,i]=S[i]
+ return U @ D @ VT
-where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
-Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
+X = np.array([ [1.0,-1.0], [1.0,-1.0]])
+#X = np.array([[1, 2], [3, 4], [5, 6]])
-Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
+print(X)
+C = SVD(X)
+# Print the difference between the original matrix and the SVD one
+print(C-X)
+
+All values beyond \( p-1 \) are all zero.
+The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first +column is the row-wise sum of the other two columns. The rank of a +matrix (the column rank) is the dimension of space spanned by the +column vectors. The rank of the matrix is the number of linearly +independent columns, in this case just \( 2 \). We see this from the +singular values when running the above code. Running the standard +inversion algorithm for matrix inversion with \( \boldsymbol{X}^T\boldsymbol{X} \) results +in the program terminating due to a singular matrix. +
diff --git a/doc/pub/week35/html/._week35-bs043.html b/doc/pub/week35/html/._week35-bs043.html index f38e34b94..7ca263705 100644 --- a/doc/pub/week35/html/._week35-bs043.html +++ b/doc/pub/week35/html/._week35-bs043.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
+The \( U \), \( S \), and \( V \) matrices returned from the svd() function +cannot be multiplied directly. +
-$$ -\boldsymbol{\Sigma}= -\begin{bmatrix} -2& 0 \\ -0 & 1 \\ -0 & 0 \\ -\end{bmatrix} -$$ +As you can see from the code, the \( S \) vector must be converted into a +diagonal matrix. This may cause a problem as the size of the matrices +do not fit the rules of matrix multiplication, where the number of +columns in a matrix must match the number of rows in the subsequent +matrix. +
-The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
- -$$ -\boldsymbol{\Sigma}= -\begin{bmatrix} -\boldsymbol{\tilde{\Sigma}}\\ -\boldsymbol{0}\\ -\end{bmatrix}, -$$ - -where
-$$ -\boldsymbol{\tilde{\Sigma}}= -\begin{bmatrix} -2& 0 \\ -0 & 1 \\ -\end{bmatrix}, -$$ - -contains only the singular values. Note also (and we will use this below) that
- -$$ -\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}= -\begin{bmatrix} -4& 0 \\ -0 & 1 \\ -\end{bmatrix}, -$$ - -which is a \( 2\times 2 \) matrix while
-$$ -\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T= -\begin{bmatrix} -4& 0 & 0\\ -0 & 1 & 0\\ -0 & 0 & 0\\ -\end{bmatrix}, -$$ - -is a \( 3\times 3 \) matrix. The last row and column of this last matrix -contain only zeros. This will have important consequences for our SVD -decomposition of the design matrix. +
If you wish to include the zero singular values, you will need to +resize the matrices and set up a diagonal matrix as done in the above +example
diff --git a/doc/pub/week35/html/._week35-bs044.html b/doc/pub/week35/html/._week35-bs044.html index d658f6d3f..e5de836c3 100644 --- a/doc/pub/week35/html/._week35-bs044.html +++ b/doc/pub/week35/html/._week35-bs044.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as
+Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
+Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
$$ -\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}. $$ -and using the orthogonality of the matrix \( \boldsymbol{U} \) we have
- +We can SVD decompose our matrix as
$$ -\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).
+where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
-We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where
+Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
+ +Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
$$ -\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. $$ -and using our SVD decomposition of \( \boldsymbol{X} \) we have
- -$$ -\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\left(\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^{2}(\boldsymbol{V}^T\right)^{-1}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, -$$ - -which gives us, using the orthogonality of the matrix \( \boldsymbol{V} \),
- -$$ -\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, -$$ - -It means that the ordinary least square model (with the optimal -parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal -transformation of the output (or target) vector \( \boldsymbol{y} \) by the -vectors of the matrix \( \boldsymbol{U} \). Note that the summation ends at -\( p-1 \), that is \( \boldsymbol{\tilde{y}}\ne \boldsymbol{y} \). We can thus not use the -orthogonality relation for the matrix \( \boldsymbol{U} \). This can already be -when we multiply the matrices \( \boldsymbol{\Sigma}^T\boldsymbol{U}^T \). -
+All values beyond \( p-1 \) are all zero.
diff --git a/doc/pub/week35/html/._week35-bs045.html b/doc/pub/week35/html/._week35-bs045.html index b3d55ab2d..a3e921925 100644 --- a/doc/pub/week35/html/._week35-bs045.html +++ b/doc/pub/week35/html/._week35-bs045.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
-Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
$$ -\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +\boldsymbol{\Sigma}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +0 & 0 \\ +\end{bmatrix} $$ -If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
+The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
+ $$ -\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. +\boldsymbol{\Sigma}= +\begin{bmatrix} +\boldsymbol{\tilde{\Sigma}}\\ +\boldsymbol{0}\\ +\end{bmatrix}, $$ -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) -with eigenvalues given by the singular values squared, that is -
+where
$$ -\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +\boldsymbol{\tilde{\Sigma}}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +\end{bmatrix}, $$ -Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have
+contains only the singular values. Note also (and we will use this below) that
+ $$ -\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}= +\begin{bmatrix} +4& 0 \\ +0 & 1 \\ +\end{bmatrix}, $$ -If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get
+which is a \( 2\times 2 \) matrix while
$$ -\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T= +\begin{bmatrix} +4& 0 & 0\\ +0 & 1 & 0\\ +0 & 0 & 0\\ +\end{bmatrix}, $$ -This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) -with eigenvalues given by the singular values squared, that is -
-$$ -\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. -$$ - -Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an -\( n\times p \) matrix. In most supervised learning cases we have that \( n -\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). -
- -In our lectures, the number of columns will -always refer to the number of features in our data set, while the -number of rows represents the number of data inputs. Note that in -other texts you may find the opposite notation. This has consequences -for the definition of for example the covariance matrix and its relation to the SVD. +
is a \( 3\times 3 \) matrix. The last row and column of this last matrix +contain only zeros. This will have important consequences for our SVD +decomposition of the design matrix.
diff --git a/doc/pub/week35/html/._week35-bs046.html b/doc/pub/week35/html/._week35-bs046.html index 342cb0303..ed2262d2f 100644 --- a/doc/pub/week35/html/._week35-bs046.html +++ b/doc/pub/week35/html/._week35-bs046.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
- -We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary -least squares is proportional to the second derivative of the cost -function, that is we have -
+The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as
$$ -\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
- -The Hessian matrix plays an important role and is defined in this course as
+and using the orthogonality of the matrix \( \boldsymbol{U} \) we have
$$ -\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -The Hessian matrix for ordinary least squares is also proportional to -the covariance matrix. This means also that we can use the SVD to find -the eigenvalues of the covariance matrix and the Hessian matrix in -terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies. +
We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).
+ +We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where
+ +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +and using our SVD decomposition of \( \boldsymbol{X} \) we have
+ +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\left(\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^{2}(\boldsymbol{V}^T\right)^{-1}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, +$$ + +which gives us, using the orthogonality of the matrix \( \boldsymbol{V} \),
+ +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, +$$ + +It means that the ordinary least square model (with the optimal +parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal +transformation of the output (or target) vector \( \boldsymbol{y} \) by the +vectors of the matrix \( \boldsymbol{U} \). Note that the summation ends at +\( p-1 \), that is \( \boldsymbol{\tilde{y}}\ne \boldsymbol{y} \). We can thus not use the +orthogonality relation for the matrix \( \boldsymbol{U} \). This can already be +when we multiply the matrices \( \boldsymbol{\Sigma}^T\boldsymbol{U}^T \).
diff --git a/doc/pub/week35/html/._week35-bs047.html b/doc/pub/week35/html/._week35-bs047.html index d4db14941..84ddca4e3 100644 --- a/doc/pub/week35/html/._week35-bs047.html +++ b/doc/pub/week35/html/._week35-bs047.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about -the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods. -
+Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
+$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ -Suppose we have defined two vectors -\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +
If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
+$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. +$$ + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is
$$ -\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ - \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ - \end{bmatrix}, +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -where for example
+Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have
$$ -\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. $$ -With this definition and recalling that the variance is defined as
+If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get
$$ -\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. $$ -we can rewrite the covariance matrix as
+This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) +with eigenvalues given by the singular values squared, that is +
$$ -\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ - \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ - \end{bmatrix}. +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. $$ -Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. -What you will find in essentially all statistics texts are equations -with a factor \( 1/(n-1) \). This is called Bessel's correction. This -method corrects the bias in the estimation of the population variance -and covariance. It also partially corrects the bias in the estimation -of the population standard deviation. If you use a library like -Scikit-Learn or nunmpy's function to calculate the covariance, this -quantity will be computed with a factor \( 1/(n-1) \). +
Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an +\( n\times p \) matrix. In most supervised learning cases we have that \( n +\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). +
+ +In our lectures, the number of columns will +always refer to the number of features in our data set, while the +number of rows represents the number of data inputs. Note that in +other texts you may find the opposite notation. This has consequences +for the definition of for example the covariance matrix and its relation to the SVD.
diff --git a/doc/pub/week35/html/._week35-bs048.html b/doc/pub/week35/html/._week35-bs048.html index 600721a8d..fa591f6ce 100644 --- a/doc/pub/week35/html/._week35-bs048.html +++ b/doc/pub/week35/html/._week35-bs048.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
The covariance takes values between zero and infinity and may thus -lead to problems with loss of numerical precision for particularly -large values. It is common to scale the covariance matrix by -introducing instead the correlation matrix defined via the so-called -correlation function +
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
+ +We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary +least squares is proportional to the second derivative of the cost +function, that is we have
$$ -\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. +\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ -The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] -\in [-1,1] \). This avoids eventual problems with too large values. We -can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) -and \( \boldsymbol{y} \) as +
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
+ +The Hessian matrix plays an important role and is defined in this course as
+ +$$ +\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. +$$ + +The Hessian matrix for ordinary least squares is also proportional to +the covariance matrix. This means also that we can use the SVD to find +the eigenvalues of the covariance matrix and the Hessian matrix in +terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
-$$ -\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ - \mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\ - \end{bmatrix}, -$$ - -In the above example this is the function we constructed using pandas.
-
-
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression -we defined the design/feature matrix \( \boldsymbol{X} \) as +
Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about +the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods.
-$$ -\boldsymbol{X}=\begin{bmatrix} -x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ -x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ -x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ -\dots & \dots & \dots & \dots \dots & \dots \\ -x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ -x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ -\end{bmatrix}, -$$ - -with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the -entries \( n \) being the row elements. -We can rewrite the design/feature matrix in terms of its column vectors as +
Suppose we have defined two vectors +\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as
$$ -\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ + \end{bmatrix}, $$ -with a given vector
+where for example
$$ -\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. +\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). $$ -With these definitions, we can now rewrite our \( 2\times 2 \) -correlation/covariance matrix in terms of a moe general design/feature -matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) -covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +
With this definition and recalling that the variance is defined as
+$$ +\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, +$$ + +we can rewrite the covariance matrix as
+$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ + \end{bmatrix}. +$$ + +Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. +What you will find in essentially all statistics texts are equations +with a factor \( 1/(n-1) \). This is called Bessel's correction. This +method corrects the bias in the estimation of the population variance +and covariance. It also partially corrects the bias in the estimation +of the population standard deviation. If you use a library like +Scikit-Learn or nunmpy's function to calculate the covariance, this +quantity will be computed with a factor \( 1/(n-1) \).
-$$ -\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} -\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ -\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ -\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ -\dots & \dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots & \dots \\ -\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\ -\end{bmatrix}, -$$ - -and the correlation matrix
-$$ -\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} -1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ -\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ -\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ -\dots & \dots & \dots & \dots & \dots & \dots \\ -\dots & \dots & \dots & \dots & \dots & \dots \\ -\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ -\end{bmatrix}, -$$ - -
-
The Numpy function np.cov calculates the covariance elements using -the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have -the exact mean values. The following simple function uses the -np.vstack function which takes each vector of dimension \( 1\times n \) -and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +
The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function
-Note that this assumes you have the features as the rows, and the inputs as columns, that is
$$ -\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ - y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ +\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. +$$ + +The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as +
+ +$$ +\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\ \end{bmatrix}, $$ -which in turn is converted into into the \( 2\times 2 \) covariance matrix -\( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate -the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy -function np.mean(x). We can also extract the eigenvalues of the -covariance matrix through the np.linalg.eig() function. -
- - - -# Importing various packages
-import numpy as np
-n = 100
-x = np.random.normal(size=n)
-print(np.mean(x))
-y = 4+3*x+np.random.normal(size=n)
-print(np.mean(y))
-W = np.vstack((x, y))
-C = np.cov(W)
-print(C)
-
-In the above example this is the function we constructed using pandas.
diff --git a/doc/pub/week35/html/._week35-bs051.html b/doc/pub/week35/html/._week35-bs051.html index 4d58521d8..d22d74d13 100644 --- a/doc/pub/week35/html/._week35-bs051.html +++ b/doc/pub/week35/html/._week35-bs051.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
The previous example can be converted into the correlation matrix by -simply scaling the matrix elements with the variances. We should also -subtract the mean values for each column. This leads to the following -code which sets up the correlations matrix for the previous example in -a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +we defined the design/feature matrix \( \boldsymbol{X} \) as
+$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ - -import numpy as np
-n = 100
-# define two vectors
-x = np.random.random(size=n)
-y = 4+3*x+np.random.normal(size=n)
-#scaling the x and y vectors
-x = x - np.mean(x)
-y = y - np.mean(y)
-variance_x = np.sum(x@x)/n
-variance_y = np.sum(y@y)/n
-print(variance_x)
-print(variance_y)
-cov_xy = np.sum(x@y)/n
-cov_xx = np.sum(x@x)/n
-cov_yy = np.sum(y@y)/n
-C = np.zeros((2,2))
-C[0,0]= cov_xx/variance_x
-C[1,1]= cov_yy/variance_y
-C[0,1]= cov_xy/np.sqrt(variance_y*variance_x)
-C[1,0]= C[0,1]
-print(C)
-
-with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +entries \( n \) being the row elements. +We can rewrite the design/feature matrix in terms of its column vectors as +
+$$ +\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, +$$ -We see that the matrix elements along the diagonal are one as they -should be and that the matrix is symmetric. Furthermore, diagonalizing -this matrix we easily see that it is a positive definite matrix. +
with a given vector
+$$ +\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. +$$ + +With these definitions, we can now rewrite our \( 2\times 2 \) +correlation/covariance matrix in terms of a moe general design/feature +matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) +covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
-The above procedure with numpy can be made more compact if we use pandas.
+$$ +\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} +\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\ +\end{bmatrix}, +$$ + +and the correlation matrix
+$$ +\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} +1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, +$$ +diff --git a/doc/pub/week35/html/._week35-bs052.html b/doc/pub/week35/html/._week35-bs052.html index a148cc62e..96d7056aa 100644 --- a/doc/pub/week35/html/._week35-bs052.html +++ b/doc/pub/week35/html/._week35-bs052.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
The Numpy function np.cov calculates the covariance elements using +the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have +the exact mean values. The following simple function uses the +np.vstack function which takes each vector of dimension \( 1\times n \) +and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +
+ +Note that this assumes you have the features as the rows, and the inputs as columns, that is
+$$ +\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ + y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ + \end{bmatrix}, +$$ + +which in turn is converted into into the \( 2\times 2 \) covariance matrix +\( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate +the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy +function np.mean(x). We can also extract the eigenvalues of the +covariance matrix through the np.linalg.eig() function. +
-We whow here how we can set up the correlation matrix using pandas, as done in this simple code
import numpy as np
-import pandas as pd
-n = 10
+ # Importing various packages
+import numpy as np
+n = 100
x = np.random.normal(size=n)
-x = x - np.mean(x)
+print(np.mean(x))
y = 4+3*x+np.random.normal(size=n)
-y = y - np.mean(y)
-# Note that we transpose the matrix in order to stay with our ordering n x p
-X = (np.vstack((x, y))).T
-print(X)
-Xpd = pd.DataFrame(X)
-print(Xpd)
-correlation_matrix = Xpd.corr()
-print(correlation_matrix)
+print(np.mean(y))
+W = np.vstack((x, y))
+C = np.cov(W)
+print(C)
-
The previous example can be converted into the correlation matrix by +simply scaling the matrix elements with the variances. We should also +subtract the mean values for each column. This leads to the following +code which sets up the correlations matrix for the previous example in +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +
@@ -330,49 +337,27 @@ MathJax.Hub.Config({# Common imports
-import numpy as np
-import pandas as pd
-
-
-def FrankeFunction(x,y):
- term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
- term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
- term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
- term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
- return term1 + term2 + term3 + term4
-
-
-def create_X(x, y, n ):
- if len(x.shape) > 1:
- x = np.ravel(x)
- y = np.ravel(y)
-
- N = len(x)
- l = int((n+1)*(n+2)/2) # Number of elements in beta
- X = np.ones((N,l))
-
- for i in range(1,n+1):
- q = int((i)*(i+1)/2)
- for k in range(i+1):
- X[:,q+k] = (x**(i-k))*(y**k)
-
- return X
-
-
-# Making meshgrid of datapoints and compute Franke's function
-n = 4
-N = 100
-x = np.sort(np.random.uniform(0, 1, N))
-y = np.sort(np.random.uniform(0, 1, N))
-z = FrankeFunction(x, y)
-X = create_X(x, y, n=n)
-
-Xpd = pd.DataFrame(X)
-# subtract the mean values and set up the covariance matrix
-Xpd = Xpd - Xpd.mean()
-covariance_matrix = Xpd.cov()
-print(covariance_matrix)
+ import numpy as np
+n = 100
+# define two vectors
+x = np.random.random(size=n)
+y = 4+3*x+np.random.normal(size=n)
+#scaling the x and y vectors
+x = x - np.mean(x)
+y = y - np.mean(y)
+variance_x = np.sum(x@x)/n
+variance_y = np.sum(y@y)/n
+print(variance_x)
+print(variance_y)
+cov_xy = np.sum(x@y)/n
+cov_xx = np.sum(x@x)/n
+cov_yy = np.sum(y@y)/n
+C = np.zeros((2,2))
+C[0,0]= cov_xx/variance_x
+C[1,1]= cov_yy/variance_y
+C[0,1]= cov_xy/np.sqrt(variance_y*variance_x)
+C[1,0]= C[0,1]
+print(C)
-
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as
-$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. -$$ +We whow here how we can set up the correlation matrix using pandas, as done in this simple code
-To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)
-$$ -\boldsymbol{X}=\begin{bmatrix} -x_{00} & x_{01}\\ -x_{10} & x_{11}\\ -\end{bmatrix}=\begin{bmatrix} -\boldsymbol{x}_{0} & \boldsymbol{x}_{1}\\ -\end{bmatrix}. -$$ + +import numpy as np
+import pandas as pd
+n = 10
+x = np.random.normal(size=n)
+x = x - np.mean(x)
+y = 4+3*x+np.random.normal(size=n)
+y = y - np.mean(y)
+# Note that we transpose the matrix in order to stay with our ordering n x p
+X = (np.vstack((x, y))).T
+print(X)
+Xpd = pd.DataFrame(X)
+print(Xpd)
+correlation_matrix = Xpd.corr()
+print(correlation_matrix)
+
+If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \))
-$$ -\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\frac{1}{n}\begin{bmatrix} -x_{00}^2+x_{10}^2 & x_{00}x_{01}+x_{10}x_{11}\\ -x_{01}x_{00}+x_{11}x_{10} & x_{01}^2+x_{11}^2\\ -\end{bmatrix}, -$$ - -which is just
-$$ -\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ - \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ - \end{bmatrix}, -$$ - -where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).
- -It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).
diff --git a/doc/pub/week35/html/._week35-bs055.html b/doc/pub/week35/html/._week35-bs055.html index e4a7a81a2..b1e81b768 100644 --- a/doc/pub/week35/html/._week35-bs055.html +++ b/doc/pub/week35/html/._week35-bs055.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
We saw earlier that
+We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as
$$ -\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
+To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)
$$ -\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, +\boldsymbol{X}=\begin{bmatrix} +x_{00} & x_{01}\\ +x_{10} & x_{11}\\ +\end{bmatrix}=\begin{bmatrix} +\boldsymbol{x}_{0} & \boldsymbol{x}_{1}\\ +\end{bmatrix}. $$ -where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
- +If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \))
$$ -\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ - 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ - 0 & 0 & \sigma_2 & \dots & 0 & 0 \\ - 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ - 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\frac{1}{n}\begin{bmatrix} +x_{00}^2+x_{10}^2 & x_{00}x_{01}+x_{10}x_{11}\\ +x_{01}x_{00}+x_{11}x_{10} & x_{01}^2+x_{11}^2\\ \end{bmatrix}, $$ -meaning we can write
+which is just
$$ -\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ + \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ + \end{bmatrix}, $$ -Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
-$$ -\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. -$$ +where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).
+It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).
diff --git a/doc/pub/week35/html/._week35-bs056.html b/doc/pub/week35/html/._week35-bs056.html index 9702ce2cc..8377a34a2 100644 --- a/doc/pub/week35/html/._week35-bs056.html +++ b/doc/pub/week35/html/._week35-bs056.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) -are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues -given by the singular values squared, that is -
+We saw earlier that
$$ -\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive -square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that -the columns of \( \boldsymbol{V} \) are the eigenvectors of -\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of -\( \boldsymbol{X} \) in a descending order, it means that the column vectors -\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). -
- -Note that these are also the eigenvectors and eigenvalues of the -Hessian matrix. Note also that the Hessian matrix we are discussing here is from a cost function defined by the mean squared error only. -
- -If we now recall the definition of the covariance matrix (not using -Bessel's correction) we have -
- +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
$$ -\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, +$$ + +where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
+ +$$ +\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \\ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +$$ + +meaning we can write
+$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +$$ + +Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
+$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( -the number of samples) are the eigenvalues of the covariance -matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square -root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is -self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the -absolute value of the eigenvalues of \( \boldsymbol{X} \). -
diff --git a/doc/pub/week35/html/._week35-bs057.html b/doc/pub/week35/html/._week35-bs057.html index c33d870d6..d1024e3af 100644 --- a/doc/pub/week35/html/._week35-bs057.html +++ b/doc/pub/week35/html/._week35-bs057.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
For \( \boldsymbol{X}\boldsymbol{X}^T \) we found
- -$$ -\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. -$$ - -Since the matrices here have dimension \( n\times n \), we have
-$$ -\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, -$$ - -leading to
-$$ -\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. -$$ - -Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem
-$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. -$$ - -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by -the non-zero singular values plus now a series of zeros. The column -vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and -measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues +given by the singular values squared, that is
-Since we will mainly be interested in the correlations among the features -of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular -values and the column vectors of \( \boldsymbol{V} \). +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ + +
In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that +the columns of \( \boldsymbol{V} \) are the eigenvectors of +\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of +\( \boldsymbol{X} \) in a descending order, it means that the column vectors +\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they +encode from the columns of \( \boldsymbol{X} \). +
+ +Note that these are also the eigenvectors and eigenvalues of the +Hessian matrix. Note also that the Hessian matrix we are discussing here is from a cost function defined by the mean squared error only. +
+ +If we now recall the definition of the covariance matrix (not using +Bessel's correction) we have +
+ +$$ +\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, +$$ + +meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +the number of samples) are the eigenvalues of the covariance +matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square +root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is +self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the +absolute value of the eigenvalues of \( \boldsymbol{X} \).
diff --git a/doc/pub/week35/html/._week35-bs058.html b/doc/pub/week35/html/._week35-bs058.html index a7342d68f..6e90d79ec 100644 --- a/doc/pub/week35/html/._week35-bs058.html +++ b/doc/pub/week35/html/._week35-bs058.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
-
For \( \boldsymbol{X}\boldsymbol{X}^T \) we found
-Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is -our optimization problem is -
$$ -{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -or we can state it as
+Since the matrices here have dimension \( n\times n \), we have
$$ -{\displaystyle \min_{\boldsymbol{\theta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2, +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -where we have used the definition of a norm-2 vector, that is
+leading to
$$ -\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$ -By minimizing the above equation with respect to the parameters -\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the -parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by -defining a new cost function to be optimized, that is +
Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem
+$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. +$$ + +It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +the non-zero singular values plus now a series of zeros. The column +vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and +measure how much correlations are contained in the rows of \( \boldsymbol{X} \).
-$$ -{\displaystyle \min_{\boldsymbol{\theta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2 -$$ - -which leads to the Ridge regression minimization problem where we -require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is -a finite number larger than zero. By defining +
Since we will mainly be interested in the correlations among the features +of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular +values and the column vectors of \( \boldsymbol{V} \).
-$$ -C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1, -$$ - -we have a new optimization equation
-$$ -{\displaystyle \min_{\boldsymbol{\theta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1 -$$ - -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
- -Here we have defined the norm-1 as
-$$ -\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. -$$ - -
-
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
- -$$ -C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, -$$ - -and -taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain -the optimal parameters +
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is
$$ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. $$ -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
- +or we can state it as
$$ -\sum_{i=0}^{p-1} \theta_i^2 \leq t, +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2, $$ -with \( t \) a finite positive number.
- -If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
+where we have used the definition of a norm-2 vector, that is
$$ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$ -In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
+By minimizing the above equation with respect to the parameters +\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is +
-When we compare this with the ordinary least squares result we have
$$ -\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2 $$ -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
+which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining +
-We see that Ridge regression is nothing but the standard OLS with a +$$ +C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1, +$$ + +
we have a new optimization equation
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
+ +Here we have defined the norm-1 as
+$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + +Ridge regression, as discussed above, is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may diff --git a/doc/pub/week35/html/._week35-bs060.html b/doc/pub/week35/html/._week35-bs060.html index 3002a4c79..3173cf60c 100644 --- a/doc/pub/week35/html/._week35-bs060.html +++ b/doc/pub/week35/html/._week35-bs060.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss how to code LASSO regression next week, when we have introduced gradient methods.
diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 6714a5fc6..badf47a47 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -133,6 +133,14 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d None, 'linear-regression-problems'), ('Fixing the singularity', 2, None, 'fixing-the-singularity'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Basic math of the SVD', 2, None, 'basic-math-of-the-svd'), ('The SVD, a Fantastic Algorithm', 2, @@ -182,10 +190,6 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'correlation-matrix-with-pandas'), - ('Correlation Matrix with Pandas and the Franke function', - 2, - None, - 'correlation-matrix-with-pandas-and-the-franke-function'), ('Rewriting the Covariance and/or Correlation Matrix', 2, None, @@ -196,14 +200,10 @@ doconce format html week35.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'and-finally-boldsymbol-x-boldsymbol-x-t'), - ('Ridge and LASSO Regression', + ('Back to Ridge and LASSO Regression', 2, None, - 'ridge-and-lasso-regression'), - ('Deriving the Ridge Regression Equations', - 2, - None, - 'deriving-the-ridge-regression-equations'), + 'back-to-ridge-and-lasso-regression'), ('Interpreting the Ridge results', 2, None, @@ -284,29 +284,29 @@ MathJax.Hub.Config({
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) is the outcome of a +output (response, target, dependent or outcome) variable \( y_i \) is the outcome of a numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we -will treat \( y_i \) as our exact value for the response variable. +will treat \( y_i \) as our exact value for the output variable.
In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
@@ -769,9 +769,9 @@ next week) with the variance of the optimal parameters important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning/data analysis problem. -v -Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). + +Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
# matrix inversion to find beta
+ # matrix inversion to find theta
# First we set up the data
import numpy as np
x = np.random.rand(100)
@@ -847,9 +847,9 @@ X[:,1] = x
X[:,2] = x**2
X[:,3] = x**3
X[:,4] = x**4
-beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
+theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
# and then make the prediction
-ytilde = X @ beta
+ytilde = X @ theta
As we will also see in the first project, this may however not the be case in general and a standard matrix inversion -algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. +algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.
There is however a way to circumvent this problem and also @@ -1632,10 +1632,237 @@ $$
where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later.
+Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +
+
+$$
+{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
+$$
+
+
+
or we can state it as
+
+$$
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
+$$
+
+
+
where we have used the definition of a norm-2 vector, that is
+
+$$
+\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
+$$
+
+
+
By minimizing the above equation with respect to the parameters +\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is +
+ +
+$$
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
+$$
+
+
+
which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining +
+ +
+$$
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
+$$
+
+
+
we have a new optimization equation
+
+$$
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
+$$
+
+
+
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
+ +Here we have defined the norm-1 as
+
+$$
+\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert.
+$$
+
+
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
+ +
+$$
+C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
+$$
+
+
+
and +taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain +the optimal parameters +
+
+$$
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+$$
+
+
+
with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
+ +
+$$
+\sum_{i=0}^{p-1} \theta_i^2 \leq t,
+$$
+
+
+
with \( t \) a finite positive number.
+ +If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
+
+$$
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+$$
+
+
+
In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
+ +When we compare this with the ordinary least squares result we have
+
+$$
+\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+$$
+
+
+
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
+ +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +
+ +When we have discussed the singular value decomposition of the design +matrix \( \boldsymbol{X} \), we will in turn perform a more rigorous mathematical +discussion of Ridge regression. +
+ +The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn.
+ + + +import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
+
+def MSE(y_data,y_model):
+ n = np.size(y_model)
+ return np.sum((y_data-y_model)**2)/n
+
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+n = 100
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
+
+Maxpolydegree = 20
+X = np.zeros((n,Maxpolydegree))
+#We include explicitely the intercept column
+for degree in range(Maxpolydegree):
+ X[:,degree] = x**degree
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+p = Maxpolydegree
+I = np.eye(p,p)
+# Decide which values of lambda to use
+nlambdas = 6
+MSEOwnRidgePredict = np.zeros(nlambdas)
+MSERidgePredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 2, nlambdas)
+for i in range(nlambdas):
+ lmb = lambdas[i]
+ OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+ # Note: we include the intercept column and no scaling
+ RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+ RegRidge.fit(X_train,y_train)
+ # and then make the prediction
+ ytildeOwnRidge = X_train @ OwnRidgeTheta
+ ypredictOwnRidge = X_test @ OwnRidgeTheta
+ ytildeRidge = RegRidge.predict(X_train)
+ ypredictRidge = RegRidge.predict(X_test)
+ MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
+ MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+ print("Theta values for own Ridge implementation")
+ print(OwnRidgeTheta)
+ print("Theta values for Scikit-Learn Ridge implementation")
+ print(RegRidge.coef_)
+ print("MSE values for own Ridge implementation")
+ print(MSEOwnRidgePredict[i])
+ print("MSE values for Scikit-Learn Ridge implementation")
+ print(MSERidgePredict[i])
+
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')
+plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
+The results here agree when we force Scikit-Learn's Ridge function to include the first column in our design matrix. +We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. +What happens if we do not include the intercept in our fit? We will discuss this in more detail next week. +
+From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +
From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only if it is a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). The matrix has then a set of eigenpairs @@ -2415,88 +2642,6 @@ correlation_matrix = Xpd.corr()
We expand this model to the Franke function discussed above.
- - -# Common imports
-import numpy as np
-import pandas as pd
-
-
-def FrankeFunction(x,y):
- term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
- term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
- term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
- term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
- return term1 + term2 + term3 + term4
-
-
-def create_X(x, y, n ):
- if len(x.shape) > 1:
- x = np.ravel(x)
- y = np.ravel(y)
-
- N = len(x)
- l = int((n+1)*(n+2)/2) # Number of elements in beta
- X = np.ones((N,l))
-
- for i in range(1,n+1):
- q = int((i)*(i+1)/2)
- for k in range(i+1):
- X[:,q+k] = (x**(i-k))*(y**k)
-
- return X
-
-
-# Making meshgrid of datapoints and compute Franke's function
-n = 4
-N = 100
-x = np.sort(np.random.uniform(0, 1, N))
-y = np.sort(np.random.uniform(0, 1, N))
-z = FrankeFunction(x, y)
-X = create_X(x, y, n=n)
-
-Xpd = pd.DataFrame(X)
-# subtract the mean values and set up the covariance matrix
-Xpd = Xpd - Xpd.mean()
-covariance_matrix = Xpd.cov()
-print(covariance_matrix)
-
-We note here that the covariance is zero for the first rows and -columns since all matrix elements in the design matrix were set to one -(we are fitting the function in terms of a polynomial of degree \( n \)). -
- -This means that the variance for these elements will be zero and will -cause problems when we set up the correlation matrix. We can simply -drop these elements and construct a correlation -matrix without these elements. -
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is @@ -2749,60 +2894,8 @@ $$ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$
-
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
- -
-$$
-C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
-$$
-
-
-
and -taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain -the optimal parameters -
-
-$$
-\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
-$$
-
-
-
with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
- -
-$$
-\sum_{i=0}^{p-1} \theta_i^2 \leq t,
-$$
-
-
-
with \( t \) a finite positive number.
- -If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
-
-$$
-\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
-$$
-
-
-
In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
- -When we compare this with the ordinary least squares result we have
-
-$$
-\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
-$$
-
-
-
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
- -We see that Ridge regression is nothing but the standard OLS with a +
Ridge regression, as discussed above, is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may @@ -2928,7 +3021,7 @@ $$ $$
-
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss how to code LASSO regression next week, when we have introduced gradient methods.
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) is the outcome of a +output (response, target, dependent or outcome) variable \( y_i \) is the outcome of a numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we -will treat \( y_i \) as our exact value for the response variable. +will treat \( y_i \) as our exact value for the output variable.
In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
@@ -753,10 +753,10 @@ next week) with the variance of the optimal parameters important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning/data analysis problem. -v -Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). +Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
+# matrix inversion to find beta
+ # matrix inversion to find theta
# First we set up the data
import numpy as np
x = np.random.rand(100)
@@ -821,9 +821,9 @@ X[:,1] = x
X[:,2] = x**2
X[:,3] = x**3
X[:,4] = x**4
-beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
+theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
# and then make the prediction
-ytilde = X @ beta
+ytilde = X @ theta
As we will also see in the first project, this may however not the be case in general and a standard matrix inversion -algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. +algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.
There is however a way to circumvent this problem and also @@ -1568,10 +1568,212 @@ $$
where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later.
+Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. +$$ + +or we can state it as
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is
+$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +By minimizing the above equation with respect to the parameters +\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is +
+ +$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2 +$$ + +which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining +
+ +$$ +C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1, +$$ + +we have a new optimization equation
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
+ +Here we have defined the norm-1 as
+$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + + +Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
+ +$$ +C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, +$$ + +and +taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain +the optimal parameters +
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
+ +$$ +\sum_{i=0}^{p-1} \theta_i^2 \leq t, +$$ + +with \( t \) a finite positive number.
+ +If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
+ +When we compare this with the ordinary least squares result we have
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
+ +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +
+ +When we have discussed the singular value decomposition of the design +matrix \( \boldsymbol{X} \), we will in turn perform a more rigorous mathematical +discussion of Ridge regression. +
+ +The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn.
+ + + +import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
+
+def MSE(y_data,y_model):
+ n = np.size(y_model)
+ return np.sum((y_data-y_model)**2)/n
+
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+n = 100
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
+
+Maxpolydegree = 20
+X = np.zeros((n,Maxpolydegree))
+#We include explicitely the intercept column
+for degree in range(Maxpolydegree):
+ X[:,degree] = x**degree
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+p = Maxpolydegree
+I = np.eye(p,p)
+# Decide which values of lambda to use
+nlambdas = 6
+MSEOwnRidgePredict = np.zeros(nlambdas)
+MSERidgePredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 2, nlambdas)
+for i in range(nlambdas):
+ lmb = lambdas[i]
+ OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+ # Note: we include the intercept column and no scaling
+ RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+ RegRidge.fit(X_train,y_train)
+ # and then make the prediction
+ ytildeOwnRidge = X_train @ OwnRidgeTheta
+ ypredictOwnRidge = X_test @ OwnRidgeTheta
+ ytildeRidge = RegRidge.predict(X_train)
+ ypredictRidge = RegRidge.predict(X_test)
+ MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
+ MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+ print("Theta values for own Ridge implementation")
+ print(OwnRidgeTheta)
+ print("Theta values for Scikit-Learn Ridge implementation")
+ print(RegRidge.coef_)
+ print("MSE values for own Ridge implementation")
+ print(MSEOwnRidgePredict[i])
+ print("MSE values for Scikit-Learn Ridge implementation")
+ print(MSERidgePredict[i])
+
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')
+plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
+The results here agree when we force Scikit-Learn's Ridge function to include the first column in our design matrix. +We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. +What happens if we do not include the intercept in our fit? We will discuss this in more detail next week. +
+From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +
From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only if it is a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). The matrix has then a set of eigenpairs @@ -2261,86 +2463,6 @@ correlation_matrix = Xpd.corr()
We expand this model to the Franke function discussed above.
- -# Common imports
-import numpy as np
-import pandas as pd
-
-
-def FrankeFunction(x,y):
- term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
- term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
- term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
- term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
- return term1 + term2 + term3 + term4
-
-
-def create_X(x, y, n ):
- if len(x.shape) > 1:
- x = np.ravel(x)
- y = np.ravel(y)
-
- N = len(x)
- l = int((n+1)*(n+2)/2) # Number of elements in beta
- X = np.ones((N,l))
-
- for i in range(1,n+1):
- q = int((i)*(i+1)/2)
- for k in range(i+1):
- X[:,q+k] = (x**(i-k))*(y**k)
-
- return X
-
-
-# Making meshgrid of datapoints and compute Franke's function
-n = 4
-N = 100
-x = np.sort(np.random.uniform(0, 1, N))
-y = np.sort(np.random.uniform(0, 1, N))
-z = FrankeFunction(x, y)
-X = create_X(x, y, n=n)
-
-Xpd = pd.DataFrame(X)
-# subtract the mean values and set up the covariance matrix
-Xpd = Xpd - Xpd.mean()
-covariance_matrix = Xpd.cov()
-print(covariance_matrix)
-
-We note here that the covariance is zero for the first rows and -columns since all matrix elements in the design matrix were set to one -(we are fitting the function in terms of a polynomial of degree \( n \)). -
- -This means that the variance for these elements will be zero and will -cause problems when we set up the correlation matrix. We can simply -drop these elements and construct a correlation -matrix without these elements. -
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is
our optimization problem is
@@ -2544,49 +2666,7 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert.
$$
-
-
-
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
- -$$ -C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, -$$ - -and -taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain -the optimal parameters -
-$$ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
- -$$ -\sum_{i=0}^{p-1} \theta_i^2 \leq t, -$$ - -with \( t \) a finite positive number.
- -If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
-$$ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. -$$ - -In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
- -When we compare this with the ordinary least squares result we have
-$$ -\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
- -We see that Ridge regression is nothing but the standard OLS with a +
Ridge regression, as discussed above, is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may @@ -2687,7 +2767,7 @@ $$ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}. $$ -
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss how to code LASSO regression next week, when we have introduced gradient methods.
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) is the outcome of a +output (response, target, dependent or outcome) variable \( y_i \) is the outcome of a numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we -will treat \( y_i \) as our exact value for the response variable. +will treat \( y_i \) as our exact value for the output variable.
In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
@@ -830,10 +830,10 @@ next week) with the variance of the optimal parameters important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning/data analysis problem. -v -Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). +Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
+# matrix inversion to find beta
+ # matrix inversion to find theta
# First we set up the data
import numpy as np
x = np.random.rand(100)
@@ -898,9 +898,9 @@ X[:,1] =
X[:,2] = x**2
X[:,3] = x**3
X[:,4] = x**4
-beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
+theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
# and then make the prediction
-ytilde = X @ beta
+ytilde = X @ theta
As we will also see in the first project, this may however not the be case in general and a standard matrix inversion -algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. +algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.
There is however a way to circumvent this problem and also @@ -1645,10 +1645,212 @@ $$
where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later.
+Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. +$$ + +or we can state it as
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is
+$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +By minimizing the above equation with respect to the parameters +\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is +
+ +$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2 +$$ + +which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining +
+ +$$ +C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1, +$$ + +we have a new optimization equation
+$$ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
+ +Here we have defined the norm-1 as
+$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + + +Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
+ +$$ +C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, +$$ + +and +taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain +the optimal parameters +
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
+ +$$ +\sum_{i=0}^{p-1} \theta_i^2 \leq t, +$$ + +with \( t \) a finite positive number.
+ +If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
+ +When we compare this with the ordinary least squares result we have
+$$ +\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
+ +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +
+ +When we have discussed the singular value decomposition of the design +matrix \( \boldsymbol{X} \), we will in turn perform a more rigorous mathematical +discussion of Ridge regression. +
+ +The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn.
+ + + +import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
+
+def MSE(y_data,y_model):
+ n = np.size(y_model)
+ return np.sum((y_data-y_model)**2)/n
+
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+n = 100
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
+
+Maxpolydegree = 20
+X = np.zeros((n,Maxpolydegree))
+#We include explicitely the intercept column
+for degree in range(Maxpolydegree):
+ X[:,degree] = x**degree
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+p = Maxpolydegree
+I = np.eye(p,p)
+# Decide which values of lambda to use
+nlambdas = 6
+MSEOwnRidgePredict = np.zeros(nlambdas)
+MSERidgePredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 2, nlambdas)
+for i in range(nlambdas):
+ lmb = lambdas[i]
+ OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+ # Note: we include the intercept column and no scaling
+ RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+ RegRidge.fit(X_train,y_train)
+ # and then make the prediction
+ ytildeOwnRidge = X_train @ OwnRidgeTheta
+ ypredictOwnRidge = X_test @ OwnRidgeTheta
+ ytildeRidge = RegRidge.predict(X_train)
+ ypredictRidge = RegRidge.predict(X_test)
+ MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
+ MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+ print("Theta values for own Ridge implementation")
+ print(OwnRidgeTheta)
+ print("Theta values for Scikit-Learn Ridge implementation")
+ print(RegRidge.coef_)
+ print("MSE values for own Ridge implementation")
+ print(MSEOwnRidgePredict[i])
+ print("MSE values for Scikit-Learn Ridge implementation")
+ print(MSERidgePredict[i])
+
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')
+plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
+The results here agree when we force Scikit-Learn's Ridge function to include the first column in our design matrix. +We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. +What happens if we do not include the intercept in our fit? We will discuss this in more detail next week. +
+From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +
From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only if it is
a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \)
we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \).
The matrix has then a set of eigenpairs
@@ -2338,86 +2540,6 @@ correlation_matrix = XpdCorrelation Matrix with Pandas and the Franke function
-
-
-
- We note here that the covariance is zero for the first rows and
-columns since all matrix elements in the design matrix were set to one
-(we are fitting the function in terms of a polynomial of degree \( n \)).
- This means that the variance for these elements will be zero and will
-cause problems when we set up the correlation matrix. We can simply
-drop these elements and construct a correlation
-matrix without these elements.
-# Common imports
-import numpy as np
-import pandas as pd
-
-
-def FrankeFunction(x,y):
- term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
- term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
- term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
- term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
- return term1 + term2 + term3 + term4
-
-
-def create_X(x, y, n ):
- if len(x.shape) > 1:
- x = np.ravel(x)
- y = np.ravel(y)
-
- N = len(x)
- l = int((n+1)*(n+2)/2) # Number of elements in beta
- X = np.ones((N,l))
-
- for i in range(1,n+1):
- q = int((i)*(i+1)/2)
- for k in range(i+1):
- X[:,q+k] = (x**(i-k))*(y**k)
-
- return X
-
-
-# Making meshgrid of datapoints and compute Franke's function
-n = 4
-N = 100
-x = np.sort(np.random.uniform(0, 1, N))
-y = np.sort(np.random.uniform(0, 1, N))
-z = FrankeFunction(x, y)
-X = create_X(x, y, n=n)
-
-Xpd = pd.DataFrame(X)
-# subtract the mean values and set up the covariance matrix
-Xpd = Xpd - Xpd.mean()
-covariance_matrix = Xpd.cov()
-print(covariance_matrix)
-
-
Rewriting the Covariance and/or Correlation Matrix
@@ -2568,7 +2690,7 @@ values and the column vectors of \( \boldsymbol{V} \).
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is
our optimization problem is
@@ -2621,49 +2743,7 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert.
$$
-
-
-
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
- -$$ -C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, -$$ - -and -taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain -the optimal parameters -
-$$ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
- -$$ -\sum_{i=0}^{p-1} \theta_i^2 \leq t, -$$ - -with \( t \) a finite positive number.
- -If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
-$$ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. -$$ - -In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
- -When we compare this with the ordinary least squares result we have
-$$ -\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
- -We see that Ridge regression is nothing but the standard OLS with a +
Ridge regression, as discussed above, is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may @@ -2764,7 +2844,7 @@ $$ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}. $$ -
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss how to code LASSO regression next week, when we have introduced gradient methods.