From 977be1cf3e61fa2aa6109b9d5b19b9f15bb2da5c Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Sun, 5 Sep 2021 21:49:06 +0200 Subject: [PATCH] adding stuff to week 35 --- doc/pub/week35/html/week35-bs.html | 45 ++-- doc/pub/week35/html/week35-reveal.html | 160 ++++++++++++-- doc/pub/week35/html/week35-solarized.html | 149 +++++++++++-- doc/pub/week35/html/week35.html | 149 +++++++++++-- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 215 +++++++++++++++++-- doc/src/week35/week35.do.txt | 141 ++++++++++-- 7 files changed, 742 insertions(+), 117 deletions(-) diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 96b152e27..c8dc8f1b6 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -183,6 +183,11 @@ Automatically generated HTML file from DocOnce source None, 'note-about-svd-calculations'), ('Friday September 3', 2, None, 'friday-september-3'), + ('Matheamtics of the SVD and implications', + 2, + None, + 'matheamtics-of-the-svd-and-implications'), + ('Example Matrix', 2, None, 'example-matrix'), ('Ridge and LASSO Regression', 2, None, @@ -323,23 +328,25 @@ MathJax.Hub.Config({
  • Codes for the SVD
  • Note about SVD Calculations
  • Friday September 3
    • -
  • Ridge and LASSO Regression
    • -
  • More on Ridge Regression
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • A better understanding of regularization
    • -
  • Decomposing the OLS and Ridge expressions
    • -
  • Introducing the Covariance and Correlation functions
    • -
  • Correlation Function and Design/Feature Matrix
    • -
  • Covariance Matrix Examples
    • -
  • Correlation Matrix
    • -
  • Correlation Matrix with Pandas
    • -
  • Correlation Matrix with Pandas and the Franke function
    • -
  • Rewriting the Covariance and/or Correlation Matrix
    • -
  • Mathematical Properties
    • -
  • Exercises for week 36, September 6-10
    • -
  • Exercise 1: Adding Ridge and Lasso Regression
    • -
  •    Exercise: Linear Regression for a two-dimensional function
    • +
  • Matheamtics of the SVD and implications
    • +
  • Example Matrix
    • +
  • Ridge and LASSO Regression
    • +
  • More on Ridge Regression
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • A better understanding of regularization
    • +
  • Decomposing the OLS and Ridge expressions
    • +
  • Introducing the Covariance and Correlation functions
    • +
  • Correlation Function and Design/Feature Matrix
    • +
  • Covariance Matrix Examples
    • +
  • Correlation Matrix
    • +
  • Correlation Matrix with Pandas
    • +
  • Correlation Matrix with Pandas and the Franke function
    • +
  • Rewriting the Covariance and/or Correlation Matrix
    • +
  • Mathematical Properties
    • +
  • Exercises for week 36, September 6-10
    • +
  • Exercise 1: Adding Ridge and Lasso Regression
    • +
  •    Exercise: Linear Regression for a two-dimensional function
    • @@ -374,7 +381,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 4, 2021

    +

    Sep 5, 2021


    @@ -398,7 +405,7 @@ MathJax.Hub.Config({

  • 9
  • 10
  • ...
    • -
  • 63
    • +
  • 65
  • »
    • diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index bd74fb666..ffd5f8a81 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

     
    -

    Sep 4, 2021

    +

    Sep 5, 2021


    @@ -1656,8 +1656,7 @@ This serves also as a useful test of our codes.

    The examples we have looked at so far are cases where we normally can -invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we -did both for the masses and the fitting of various functions leads to +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky @@ -1670,7 +1669,9 @@ however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.

    -There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +There is however a way to circumvent this problem and also +gain some insights about the ordinary least squares approach, and +later shrinkage methods like Ridge and Lasso regressions.

    This is given by the Singular Value Decomposition (SVD) algorithm, @@ -1680,13 +1681,13 @@ swath oc applications and the decomposition is always stable numerically.

    -In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. -Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. - -

    -Let us look at a -different example where we may have problems with the standard matrix -inversion algorithm. Thereafter we dive into the math of the SVD. +In machine learning it plays a central role in dealing with for +example design matrices that may be near singular or singular. +Furthermore, as we will see here, the singular values can be related +to the covariance matrix (and thereby the correlation matrix) and in +turn the variance of a given quantity. It plays also an important role +in the principal component analysis where high-dimensional data can be +reduced to the statistically relevant features. @@ -1916,7 +1917,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des 

    import numpy as np
 # SVD inversion
-def SVDinv(A):
+def SVD(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
     SVD is numerically more stable than the inversion algorithms provided by
     numpy and scipy.linalg at the cost of being slower.
@@ -1940,7 +1941,7 @@ X = np.array([ [1.0,-#X = np.array([[1, 2], [3, 4], [5, 6]])
 
 print(X)
-C = SVDinv(X)
+C = SVD(X)
 # Print the difference between the original matrix and the SVD one
 print(C-X)
    @@ -1964,15 +1965,16 @@ The \( U \), \( S \), and \( V \) matrices returned from the svd() functi cannot be multiplied directly.

    -As you can see from the code, the \( S \) -vector must be converted into a diagonal matrix. This may cause a -as -the size of the matrices do not fit the rules of matrix -multiplication, where the number of columns in a matrix must match the -number of rows in the subsequent matrix. +As you can see from the code, the \( S \) vector must be converted into a +diagonal matrix. This may cause a problem as the size of the matrices +do not fit the rules of matrix multiplication, where the number of +columns in a matrix must match the number of rows in the subsequent +matrix.

    -If you wish to include the zero singular values, you will need to resize the matrices. More about this later. +If you wish to include the zero singular values, you will need to +resize the matrices and set up a diagonal matrix as done in the above +example @@ -1987,6 +1989,124 @@ More material will be added here, see handwritten notes also. Note that this m +

    +

    Matheamtics of the SVD and implications

    + +

    +Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies. + +

    +Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \) +

     
    +$$ +\boldsymbol{X}= +\begin{bmatrix} +x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\ +x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\ +x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\ +\end{bmatrix} +$$ +

     
    + +

    +We can SVD decompose our matrix as +

     
    +$$ +\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ +

     
    + +where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \). + +

    +Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \). + +

    +Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is + +

     
    +$$ +\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. +$$ +

     
    + +

    +All values beyond \( p-1 \) are all zero. +

    + + +
    +

    Example Matrix

    + +

    +As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \) + +

     
    +$$ +\boldsymbol{\Sigma}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +0 & 0 \\ +\end{bmatrix} +$$ +

     
    + +

    +The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as + +

     
    +$$ +\boldsymbol{\Sigma}= +\begin{bmatrix} +\boldsymbol{\tilde{\Sigma}}\\ +\boldsymbol{0}\\ +\end{bmatrix}, +$$ +

     
    + +

    +where +

     
    +$$ +\boldsymbol{\tilde{\Sigma}}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +\end{bmatrix}, +$$ +

     
    + +contains only the singular values. Note also (and we will use this below) that + +

     
    +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\sigma}= +\begin{bmatrix} +4& 0 \\ +0 & 1 \\ +\end{bmatrix}, +$$ +

     
    + +which is a \( 2\times 2 \) matrix while +

     
    +$$ +\boldsymbol{\Sigma}\boldsymbol{\sigma}^T= +\begin{bmatrix} +4& 0 & 0\\ +0 & 1 & 0\\ +0 & 0 & 0\\ +\end{bmatrix}, +$$ +

     
    + +is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix. +

    + +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index d76aa761e..fc04fd44f 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -203,6 +203,11 @@ div { text-align: justify; text-justify: inter-word; } None, 'note-about-svd-calculations'), ('Friday September 3', 2, None, 'friday-september-3'), + ('Matheamtics of the SVD and implications', + 2, + None, + 'matheamtics-of-the-svd-and-implications'), + ('Example Matrix', 2, None, 'example-matrix'), ('Ridge and LASSO Regression', 2, None, @@ -300,7 +305,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 4, 2021

    +

    Sep 5, 2021












    @@ -1735,8 +1740,7 @@ This serves also as a useful test of our codes.

    The examples we have looked at so far are cases where we normally can -invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we -did both for the masses and the fitting of various functions leads to +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky @@ -1749,7 +1753,9 @@ however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.

    -There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +There is however a way to circumvent this problem and also +gain some insights about the ordinary least squares approach, and +later shrinkage methods like Ridge and Lasso regressions.

    This is given by the Singular Value Decomposition (SVD) algorithm, @@ -1759,13 +1765,13 @@ swath oc applications and the decomposition is always stable numerically.

    -In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. -Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. - -

    -Let us look at a -different example where we may have problems with the standard matrix -inversion algorithm. Thereafter we dive into the math of the SVD. +In machine learning it plays a central role in dealing with for +example design matrices that may be near singular or singular. +Furthermore, as we will see here, the singular values can be related +to the covariance matrix (and thereby the correlation matrix) and in +turn the variance of a given quantity. It plays also an important role +in the principal component analysis where high-dimensional data can be +reduced to the statistically relevant features. @@ -1976,7 +1982,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des 

    import numpy as np
 # SVD inversion
-def SVDinv(A):
+def SVD(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
     SVD is numerically more stable than the inversion algorithms provided by
     numpy and scipy.linalg at the cost of being slower.
@@ -2000,7 +2006,7 @@ X = np.array([ [1.0,-#X = np.array([[1, 2], [3, 4], [5, 6]])
 
 print(X)
-C = SVDinv(X)
+C = SVD(X)
 # Print the difference between the original matrix and the SVD one
 print(C-X)
    @@ -2024,15 +2030,16 @@ The \( U \), \( S \), and \( V \) matrices returned from the svd() functi cannot be multiplied directly.

    -As you can see from the code, the \( S \) -vector must be converted into a diagonal matrix. This may cause a -as -the size of the matrices do not fit the rules of matrix -multiplication, where the number of columns in a matrix must match the -number of rows in the subsequent matrix. +As you can see from the code, the \( S \) vector must be converted into a +diagonal matrix. This may cause a problem as the size of the matrices +do not fit the rules of matrix multiplication, where the number of +columns in a matrix must match the number of rows in the subsequent +matrix.

    -If you wish to include the zero singular values, you will need to resize the matrices. More about this later. +If you wish to include the zero singular values, you will need to +resize the matrices and set up a diagonal matrix as done in the above +example











    @@ -2048,6 +2055,108 @@ More material will be added here, see handwritten notes also. Note that this m











    +

    Matheamtics of the SVD and implications

    + +

    +Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies. + +

    +Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \) +$$ +\boldsymbol{X}= +\begin{bmatrix} +x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\ +x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\ +x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\ +\end{bmatrix} +$$ + +

    +We can SVD decompose our matrix as +$$ +\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \). + +

    +Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \). + +

    +Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is + +$$ +\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. +$$ + +

    +All values beyond \( p-1 \) are all zero. + +

    +









    + +

    Example Matrix

    + +

    +As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \) + +$$ +\boldsymbol{\Sigma}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +0 & 0 \\ +\end{bmatrix} +$$ + +

    +The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as + +$$ +\boldsymbol{\Sigma}= +\begin{bmatrix} +\boldsymbol{\tilde{\Sigma}}\\ +\boldsymbol{0}\\ +\end{bmatrix}, +$$ + +

    +where +$$ +\boldsymbol{\tilde{\Sigma}}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +\end{bmatrix}, +$$ + +contains only the singular values. Note also (and we will use this below) that + +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\sigma}= +\begin{bmatrix} +4& 0 \\ +0 & 1 \\ +\end{bmatrix}, +$$ + +which is a \( 2\times 2 \) matrix while +$$ +\boldsymbol{\Sigma}\boldsymbol{\sigma}^T= +\begin{bmatrix} +4& 0 & 0\\ +0 & 1 & 0\\ +0 & 0 & 0\\ +\end{bmatrix}, +$$ + +is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix. + +

    +









    +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index db79e88fa..3d05a9b45 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -208,6 +208,11 @@ div { text-align: justify; text-justify: inter-word; } None, 'note-about-svd-calculations'), ('Friday September 3', 2, None, 'friday-september-3'), + ('Matheamtics of the SVD and implications', + 2, + None, + 'matheamtics-of-the-svd-and-implications'), + ('Example Matrix', 2, None, 'example-matrix'), ('Ridge and LASSO Regression', 2, None, @@ -305,7 +310,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 4, 2021

    +

    Sep 5, 2021












    @@ -1740,8 +1745,7 @@ This serves also as a useful test of our codes.

    The examples we have looked at so far are cases where we normally can -invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we -did both for the masses and the fitting of various functions leads to +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky @@ -1754,7 +1758,9 @@ however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.

    -There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +There is however a way to circumvent this problem and also +gain some insights about the ordinary least squares approach, and +later shrinkage methods like Ridge and Lasso regressions.

    This is given by the Singular Value Decomposition (SVD) algorithm, @@ -1764,13 +1770,13 @@ swath oc applications and the decomposition is always stable numerically.

    -In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. -Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. - -

    -Let us look at a -different example where we may have problems with the standard matrix -inversion algorithm. Thereafter we dive into the math of the SVD. +In machine learning it plays a central role in dealing with for +example design matrices that may be near singular or singular. +Furthermore, as we will see here, the singular values can be related +to the covariance matrix (and thereby the correlation matrix) and in +turn the variance of a given quantity. It plays also an important role +in the principal component analysis where high-dimensional data can be +reduced to the statistically relevant features. @@ -1981,7 +1987,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des 

    import numpy as np
 # SVD inversion
-def SVDinv(A):
+def SVD(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
     SVD is numerically more stable than the inversion algorithms provided by
     numpy and scipy.linalg at the cost of being slower.
@@ -2005,7 +2011,7 @@ X = np.a
 #X = np.array([[1, 2], [3, 4], [5, 6]])
 
 print(X)
-C = SVDinv(X)
+C = SVD(X)
 # Print the difference between the original matrix and the SVD one
 print(C-X)
    @@ -2029,15 +2035,16 @@ The \( U \), \( S \), and \( V \) matrices returned from the svd() functi cannot be multiplied directly.

    -As you can see from the code, the \( S \) -vector must be converted into a diagonal matrix. This may cause a -as -the size of the matrices do not fit the rules of matrix -multiplication, where the number of columns in a matrix must match the -number of rows in the subsequent matrix. +As you can see from the code, the \( S \) vector must be converted into a +diagonal matrix. This may cause a problem as the size of the matrices +do not fit the rules of matrix multiplication, where the number of +columns in a matrix must match the number of rows in the subsequent +matrix.

    -If you wish to include the zero singular values, you will need to resize the matrices. More about this later. +If you wish to include the zero singular values, you will need to +resize the matrices and set up a diagonal matrix as done in the above +example











    @@ -2053,6 +2060,108 @@ More material will be added here, see handwritten notes also. Note that this m











    +

    Matheamtics of the SVD and implications

    + +

    +Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies. + +

    +Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \) +$$ +\boldsymbol{X}= +\begin{bmatrix} +x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\ +x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\ +x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\ +\end{bmatrix} +$$ + +

    +We can SVD decompose our matrix as +$$ +\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \). + +

    +Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \). + +

    +Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is + +$$ +\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. +$$ + +

    +All values beyond \( p-1 \) are all zero. + +

    +









    + +

    Example Matrix

    + +

    +As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \) + +$$ +\boldsymbol{\Sigma}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +0 & 0 \\ +\end{bmatrix} +$$ + +

    +The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as + +$$ +\boldsymbol{\Sigma}= +\begin{bmatrix} +\boldsymbol{\tilde{\Sigma}}\\ +\boldsymbol{0}\\ +\end{bmatrix}, +$$ + +

    +where +$$ +\boldsymbol{\tilde{\Sigma}}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +\end{bmatrix}, +$$ + +contains only the singular values. Note also (and we will use this below) that + +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\sigma}= +\begin{bmatrix} +4& 0 \\ +0 & 1 \\ +\end{bmatrix}, +$$ + +which is a \( 2\times 2 \) matrix while +$$ +\boldsymbol{\Sigma}\boldsymbol{\sigma}^T= +\begin{bmatrix} +4& 0 & 0\\ +0 & 1 & 0\\ +0 & 0 & 0\\ +\end{bmatrix}, +$$ + +is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix. + +

    +









    +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 9f03671e568f62441cc4e039b90932865ab10b6c..8e0cda61592490ee1faf1a7d31f94fd3a862c9df 100644 GIT binary patch literal 192 zcmV;x06+g9iwFQBA2neB1MSaC3c@fD2H>uHia9|^+N50zx^N+gc!88oZLCddlA^u6 zeSoeMH${Yei}?u|hMB`=wcc$acXz>J2q7tBFlLtWDM>uPCzNuaY^X8K7>j|>gvTrZ zvfN59opr(tt2EUal|}WgZ)hva4|C>I;F*8oSV;@peeWu*Kxv1$)-~J^>x4(L?Ntti uMmv6i!D}ZCf>1q(qL5B%B`#rW^vQ_EM&YlI@jTD-zV-k|kqUzV2mk_Wl|}V#ZWt@e4}0cU;F*8oSV;@peeWu*Kxv1$)-~J^>zGHf?Ntti uMmx5^;I)$mL8u-?QAj7X5|^+w`ea07qwv?yc%J8ZUwZ&I1jk1J2mk=HHdeO) diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 6d35e29e2..31946a305 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 4, 2021**\n", + "Date: **Sep 5, 2021**\n", "\n", "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -2052,8 +2052,7 @@ "\n", "\n", "The examples we have looked at so far are cases where we normally can\n", - "invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion as we\n", - "did both for the masses and the fitting of various functions leads to\n", + "invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion where we fit of various functions leads to\n", "row vectors of the design matrix which are essentially orthogonal due\n", "to the polynomial character of our model. Obtaining the inverse of the\n", "design matrix is then often done via a so-called LU, QR or Cholesky\n", @@ -2065,7 +2064,9 @@ "however not the be case in general and a standard matrix inversion\n", "algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.\n", "\n", - "There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. \n", + "There is however a way to circumvent this problem and also\n", + "gain some insights about the ordinary least squares approach, and\n", + "later shrinkage methods like Ridge and Lasso regressions.\n", "\n", "This is given by the **Singular Value Decomposition** (SVD) algorithm,\n", "perhaps the most powerful linear algebra algorithm. The SVD provides\n", @@ -2073,14 +2074,13 @@ "swath oc applications and the decomposition is always stable\n", "numerically.\n", "\n", - "In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.\n", - "Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. \n", - "\n", - "\n", - "\n", - "Let us look at a\n", - "different example where we may have problems with the standard matrix\n", - "inversion algorithm. Thereafter we dive into the math of the SVD.\n", + "In machine learning it plays a central role in dealing with for\n", + "example design matrices that may be near singular or singular.\n", + "Furthermore, as we will see here, the singular values can be related\n", + "to the covariance matrix (and thereby the correlation matrix) and in\n", + "turn the variance of a given quantity. It plays also an important role\n", + "in the principal component analysis where high-dimensional data can be\n", + "reduced to the statistically relevant features.\n", "\n", "\n", "\n", @@ -2380,7 +2380,7 @@ "source": [ "import numpy as np\n", "# SVD inversion\n", - "def SVDinv(A):\n", + "def SVD(A):\n", " ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).\n", " SVD is numerically more stable than the inversion algorithms provided by\n", " numpy and scipy.linalg at the cost of being slower.\n", @@ -2404,7 +2404,7 @@ "#X = np.array([[1, 2], [3, 4], [5, 6]])\n", "\n", "print(X)\n", - "C = SVDinv(X)\n", + "C = SVD(X)\n", "# Print the difference between the original matrix and the SVD one\n", "print(C-X)" ] @@ -2428,14 +2428,15 @@ "The $U$, $S$, and $V$ matrices returned from the **svd()** function\n", "cannot be multiplied directly.\n", "\n", - "As you can see from the code, the $S$\n", - "vector must be converted into a diagonal matrix. This may cause a \n", - "as\n", - "the size of the matrices do not fit the rules of matrix\n", - "multiplication, where the number of columns in a matrix must match the\n", - "number of rows in the subsequent matrix.\n", + "As you can see from the code, the $S$ vector must be converted into a\n", + "diagonal matrix. This may cause a problem as the size of the matrices\n", + "do not fit the rules of matrix multiplication, where the number of\n", + "columns in a matrix must match the number of rows in the subsequent\n", + "matrix.\n", "\n", - "If you wish to include the zero singular values, you will need to resize the matrices. More about this later.\n", + "If you wish to include the zero singular values, you will need to\n", + "resize the matrices and set up a diagonal matrix as done in the above\n", + "example\n", "\n", "\n", "\n", @@ -2448,6 +2449,178 @@ "More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at .\n", "\n", "\n", + "## Matheamtics of the SVD and implications\n", + "\n", + "Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.\n", + "\n", + "Our starting point is our design matrix $\\boldsymbol{X}$ of dimension $n\\times p$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}=\n", + "\\begin{bmatrix}\n", + "x_{0,0}& x_{0,1} &x_{0,2}& \\dots & \\dots &x_{0,p-1}\\\\\n", + "x_{1,0}& x_{1,1} &x_{1,2& \\dots & \\dots &x_{1,p-1}\\\\\n", + "x_{2,0}& x_{2,1} &x_{2,2}& \\dots & \\dots &x_{2,p-1}\\\\\n", + "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n", + "x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,p-1}\\\\\n", + "\\end{bmatrix}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can SVD decompose our matrix as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $\\boldsymbol{U}$ is an orthogonal matrix of dimension $n\\times n$, meaning that $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{I}_n$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $n \\times n$.\n", + "\n", + "Similarly, $\\boldsymbol{V}$ is an orthogonal matrix of dimension $p\\times p$, meaning that $\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{I}_p$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $p \\times p$.\n", + "\n", + "Finally $\\boldsymbol{\\Sigma}$ contains the singular values $\\sigma_i$. This matrix has dimension $n\\times p$ and the singular values $\\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\sigma_0 > \\sigma_1 > \\sigma_2 > \\dots > \\sigma_{p-1} > 0.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "All values beyond $p-1$ are all zero.\n", + "\n", + "## Example Matrix\n", + "\n", + "As an example, consider the following $3\\times 2$ example for the matrix $\\boldsymbol{\\Sigma}$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\Sigma}=\n", + "\\begin{bmatrix}\n", + "2& 0 \\\\\n", + "0 & 1 \\\\\n", + "0 & 0 \\\\\n", + "\\end{bmatrix}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The singular values are $\\sigma_0=2$ and $\\sigma_1=1$. It is common to rewrite the matrix $\\boldsymbol{\\Sigma}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\Sigma}=\n", + "\\begin{bmatrix}\n", + "\\boldsymbol{\\tilde{\\Sigma}}\\\\\n", + "\\boldsymbol{0}\\\\\n", + "\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\tilde{\\Sigma}}=\n", + "\\begin{bmatrix}\n", + "2& 0 \\\\\n", + "0 & 1 \\\\\n", + "\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "contains only the singular values. Note also (and we will use this below) that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\n", + "\\begin{bmatrix}\n", + "4& 0 \\\\\n", + "0 & 1 \\\\\n", + "\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which is a $2\\times 2 $ matrix while" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\Sigma}\\boldsymbol{\\sigma}^T=\n", + "\\begin{bmatrix}\n", + "4& 0 & 0\\\\\n", + "0 & 1 & 0\\\\\n", + "0 & 0 & 0\\\\\n", + "\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "is a $3\\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.\n", + "\n", + "\n", "## Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index 1b2d1f0a8..2fd0133d0 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -1274,8 +1274,7 @@ This serves also as a useful test of our codes. !bblock The examples we have looked at so far are cases where we normally can -invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion as we -did both for the masses and the fitting of various functions leads to +invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky @@ -1287,7 +1286,9 @@ this may however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. -There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +There is however a way to circumvent this problem and also +gain some insights about the ordinary least squares approach, and +later shrinkage methods like Ridge and Lasso regressions. This is given by the _Singular Value Decomposition_ (SVD) algorithm, perhaps the most powerful linear algebra algorithm. The SVD provides @@ -1295,14 +1296,17 @@ a numerically stable matrix decomposition that is used in a large swath oc applications and the decomposition is always stable numerically. -In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. -Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. +In machine learning it plays a central role in dealing with for +example design matrices that may be near singular or singular. +Furthermore, as we will see here, the singular values can be related +to the covariance matrix (and thereby the correlation matrix) and in +turn the variance of a given quantity. It plays also an important role +in the principal component analysis where high-dimensional data can be +reduced to the statistically relevant features. + -Let us look at a -different example where we may have problems with the standard matrix -inversion algorithm. Thereafter we dive into the math of the SVD. !eblock @@ -1494,7 +1498,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des !bc pycod import numpy as np # SVD inversion -def SVDinv(A): +def SVD(A): ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD). SVD is numerically more stable than the inversion algorithms provided by numpy and scipy.linalg at the cost of being slower. @@ -1518,7 +1522,7 @@ X = np.array([ [1.0,-1.0], [1.0,-1.0]]) #X = np.array([[1, 2], [3, 4], [5, 6]]) print(X) -C = SVDinv(X) +C = SVD(X) # Print the difference between the original matrix and the SVD one print(C-X) !ec @@ -1539,14 +1543,15 @@ in the program terminating due to a singular matrix. The $U$, $S$, and $V$ matrices returned from the _svd()_ function cannot be multiplied directly. -As you can see from the code, the $S$ -vector must be converted into a diagonal matrix. This may cause a -as -the size of the matrices do not fit the rules of matrix -multiplication, where the number of columns in a matrix must match the -number of rows in the subsequent matrix. +As you can see from the code, the $S$ vector must be converted into a +diagonal matrix. This may cause a problem as the size of the matrices +do not fit the rules of matrix multiplication, where the number of +columns in a matrix must match the number of rows in the subsequent +matrix. -If you wish to include the zero singular values, you will need to resize the matrices. More about this later. +If you wish to include the zero singular values, you will need to +resize the matrices and set up a diagonal matrix as done in the above +example @@ -1560,6 +1565,108 @@ If you wish to include the zero singular values, you will need to resize the mat More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at URL:"https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021". +!split +===== Matheamtics of the SVD and implications ===== + +Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies. + +Our starting point is our design matrix $\bm{X}$ of dimension $n\times p$ +!bt +\[ +\bm{X}= +\begin{bmatrix} +x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\ +x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\ +x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\ +\end{bmatrix} +\] +!et + +We can SVD decompose our matrix as +!bt +\[ +\bm{X}=\bm{U}\bm{\Sigma}\bm{V}^T, +\] +!et +where $\bm{U}$ is an orthogonal matrix of dimension $n\times n$, meaning that $\bm{U}\bm{U}^T=\bm{U}^T\bm{U}=\bm{I}_n$. Here $\bm{I}_n$ is the unit matrix of dimension $n \times n$. + +Similarly, $\bm{V}$ is an orthogonal matrix of dimension $p\times p$, meaning that $\bm{V}\bm{V}^T=\bm{V}^T\bm{V}=\bm{I}_p$. Here $\bm{I}_n$ is the unit matrix of dimension $p \times p$. + +Finally $\bm{\Sigma}$ contains the singular values $\sigma_i$. This matrix has dimension $n\times p$ and the singular values $\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is + +!bt +\[ +\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. +\] +!et + +All values beyond $p-1$ are all zero. + +!split +===== Example Matrix ===== + +As an example, consider the following $3\times 2$ example for the matrix $\bm{\Sigma}$ + +!bt +\[ +\bm{\Sigma}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +0 & 0 \\ +\end{bmatrix} +\] +!et + +The singular values are $\sigma_0=2$ and $\sigma_1=1$. It is common to rewrite the matrix $\bm{\Sigma}$ as + +!bt +\[ +\bm{\Sigma}= +\begin{bmatrix} +\bm{\tilde{\Sigma}}\\ +\bm{0}\\ +\end{bmatrix}, +\] +!et + +where +!bt +\[ +\bm{\tilde{\Sigma}}= +\begin{bmatrix} +2& 0 \\ +0 & 1 \\ +\end{bmatrix}, +\] +!et +contains only the singular values. Note also (and we will use this below) that + +!bt +\[ +\bm{\Sigma}^T\bm{\sigma}= +\begin{bmatrix} +4& 0 \\ +0 & 1 \\ +\end{bmatrix}, +\] +!et +which is a $2\times 2 $ matrix while +!bt +\[ +\bm{\Sigma}\bm{\sigma}^T= +\begin{bmatrix} +4& 0 & 0\\ +0 & 1 & 0\\ +0 & 0 & 0\\ +\end{bmatrix}, +\] +!et +is a $3\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix. + + !split ===== Ridge and LASSO Regression =====