From 28a9f6ee81415c952b9af933466bb4b7d67ca023 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 6 Sep 2021 16:08:28 +0200 Subject: [PATCH] updating week 35 --- doc/pub/week35/html/week35-bs.html | 41 ++++--- doc/pub/week35/html/week35-reveal.html | 73 +++++++++-- doc/pub/week35/html/week35-solarized.html | 66 ++++++++-- doc/pub/week35/html/week35.html | 66 ++++++++-- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 122 +++++++++++++++++-- doc/src/week35/week35.do.txt | 65 ++++++++-- 7 files changed, 375 insertions(+), 58 deletions(-) diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 2d850f8d1..10c7c3629 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -204,6 +204,10 @@ Automatically generated HTML file from DocOnce source 2, None, 'introducing-the-covariance-and-correlation-functions'), + ('Covariance and Correlation Matrix', + 2, + None, + 'covariance-and-correlation-matrix'), ('Correlation Function and Design/Feature Matrix', 2, None, @@ -225,6 +229,7 @@ Automatically generated HTML file from DocOnce source 2, None, 'rewriting-the-covariance-and-or-correlation-matrix'), + ('Linking with the SVD', 2, None, 'linking-with-the-svd'), ('Ridge and LASSO Regression', 2, None, @@ -346,22 +351,24 @@ MathJax.Hub.Config({
  • Further properties (important for our analyses later)
  • Meet the Covariance Matrix
  • Introducing the Covariance and Correlation functions
    • -
  • Correlation Function and Design/Feature Matrix
    • -
  • Covariance Matrix Examples
    • -
  • Correlation Matrix
    • -
  • Correlation Matrix with Pandas
    • -
  • Correlation Matrix with Pandas and the Franke function
    • -
  • Rewriting the Covariance and/or Correlation Matrix
    • -
  • Ridge and LASSO Regression
    • -
  • More on Ridge Regression
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • A better understanding of regularization
    • -
  • Decomposing the OLS and Ridge expressions
    • -
  • Mathematical Properties
    • -
  • Exercises for week 36, September 6-10
    • -
  • Exercise 1: Adding Ridge and Lasso Regression
    • -
  •    Exercise: Linear Regression for a two-dimensional function
    • +
  • Covariance and Correlation Matrix
    • +
  • Correlation Function and Design/Feature Matrix
    • +
  • Covariance Matrix Examples
    • +
  • Correlation Matrix
    • +
  • Correlation Matrix with Pandas
    • +
  • Correlation Matrix with Pandas and the Franke function
    • +
  • Rewriting the Covariance and/or Correlation Matrix
    • +
  • Linking with the SVD
    • +
  • Ridge and LASSO Regression
    • +
  • More on Ridge Regression
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • A better understanding of regularization
    • +
  • Decomposing the OLS and Ridge expressions
    • +
  • Mathematical Properties
    • +
  • Exercises for week 36, September 6-10
    • +
  • Exercise 1: Adding Ridge and Lasso Regression
    • +
  •    Exercise: Linear Regression for a two-dimensional function
    • @@ -420,7 +427,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 68
    • +
  • 70
  • »
    • diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 57202077c..873bd62d3 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -2340,6 +2340,11 @@ and covariance. It also partially corrects the bias in the estimation of the population standard deviation. If you use a library like Scikit-Learn or nunmpy's function calculate the covariance, this quantity will be computed with a factor \( 1/(n-1) \). + + + +
    +

    Covariance and Correlation Matrix

    The covariance takes values between zero and infinity and may thus @@ -2411,7 +2416,7 @@ $$

    With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) @@ -2454,14 +2459,12 @@ the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    +Note that this assumes you have the features as the rows, and the inputs as columns, that is

     
    $$ -\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\ - x_1 & y_1 \\ - x_2 & y_2\\ - \dots & \dots \\ - x_{n-2} & y_{n-2}\\ - x_{n-1} & y_{n-1} & +\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ + y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$

     
    @@ -2550,6 +2553,7 @@ x = np.random.normal(size=n) x = x - np.mean(x) y = 4+3*x+np.random.normal(size=n) y = y - np.mean(y) +# Note that we transpose the matrix in order to stay with our ordering n x p X = (np.vstack((x, y))).T print(X) Xpd = pd.DataFrame(X) @@ -2678,6 +2682,61 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t

    +
    +

    Linking with the SVD

    + +

    +We saw earlier that +

     
    +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ +

     
    + +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +

     
    +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, +$$ +

     
    + +where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is + +

     
    +$$ +\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +$$ +

     
    + +meaning we can write +

     
    +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +$$ +

     
    + +Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

     
    +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. +$$ +

     
    + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +

     
    +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ +

     
    +

    + +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index 4c3ca31e1..bc31b912f 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -224,6 +224,10 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'introducing-the-covariance-and-correlation-functions'), + ('Covariance and Correlation Matrix', + 2, + None, + 'covariance-and-correlation-matrix'), ('Correlation Function and Design/Feature Matrix', 2, None, @@ -245,6 +249,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'rewriting-the-covariance-and-or-correlation-matrix'), + ('Linking with the SVD', 2, None, 'linking-with-the-svd'), ('Ridge and LASSO Regression', 2, None, @@ -2364,6 +2369,11 @@ of the population standard deviation. If you use a library like Scikit-Learn or nunmpy's function calculate the covariance, this quantity will be computed with a factor \( 1/(n-1) \). +

    +









    + +

    Covariance and Correlation Matrix

    +

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly @@ -2424,7 +2434,7 @@ $$

    With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) @@ -2463,13 +2473,11 @@ the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    +Note that this assumes you have the features as the rows, and the inputs as columns, that is $$ -\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\ - x_1 & y_1 \\ - x_2 & y_2\\ - \dots & \dots \\ - x_{n-2} & y_{n-2}\\ - x_{n-1} & y_{n-1} & +\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ + y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$ @@ -2556,6 +2564,7 @@ x = np.random.normal(size=n) x = x - np.mean(x) y = 4+3*x+np.random.normal(size=n) y = y - np.mean(y) +# Note that we transpose the matrix in order to stay with our ordering n x p X = (np.vstack((x, y))).T print(X) Xpd = pd.DataFrame(X) @@ -2675,6 +2684,49 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t











    +

    Linking with the SVD

    + +

    +We saw earlier that +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, +$$ + +where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is + +$$ +\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +$$ + +meaning we can write +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +$$ + +Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. +$$ + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ + +

    +









    +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 4c69e39eb..b261f20e9 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -229,6 +229,10 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'introducing-the-covariance-and-correlation-functions'), + ('Covariance and Correlation Matrix', + 2, + None, + 'covariance-and-correlation-matrix'), ('Correlation Function and Design/Feature Matrix', 2, None, @@ -250,6 +254,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'rewriting-the-covariance-and-or-correlation-matrix'), + ('Linking with the SVD', 2, None, 'linking-with-the-svd'), ('Ridge and LASSO Regression', 2, None, @@ -2369,6 +2374,11 @@ of the population standard deviation. If you use a library like Scikit-Learn or nunmpy's function calculate the covariance, this quantity will be computed with a factor \( 1/(n-1) \). +

    +









    + +

    Covariance and Correlation Matrix

    +

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly @@ -2429,7 +2439,7 @@ $$

    With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) @@ -2468,13 +2478,11 @@ the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    +Note that this assumes you have the features as the rows, and the inputs as columns, that is $$ -\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\ - x_1 & y_1 \\ - x_2 & y_2\\ - \dots & \dots \\ - x_{n-2} & y_{n-2}\\ - x_{n-1} & y_{n-1} & +\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ + y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$ @@ -2561,6 +2569,7 @@ x = np.r x = x - np.mean(x) y = 4+3*x+np.random.normal(size=n) y = y - np.mean(y) +# Note that we transpose the matrix in order to stay with our ordering n x p X = (np.vstack((x, y))).T print(X) Xpd = pd.DataFrame(X) @@ -2680,6 +2689,49 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t











    +

    Linking with the SVD

    + +

    +We saw earlier that +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, +$$ + +where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is + +$$ +\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +$$ + +meaning we can write +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +$$ + +Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. +$$ + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ + +

    +









    +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 21138d176c700d39729d11f9fda6434286352dfb..0978c9ee2cc5d88b22010a1cca7cd476b9eecfdc 100644 GIT binary patch delta 157 zcmV;O0Al~Z0l)zpABzY8&>%Kp00ZsMOA5j;5C-6`dx|+hNt(2g8!UOsIgGhVlPO6& zzbBL?K>1L4Ojt~SP?k^@09kIOm(Dt2hEV*Eh74<%c=*De%lcajc|;?Y?)F zR-m-QT@k00;m8@v}z* delta 157 zcmV;O0Al~Z0l)zpABzY8b{aNe00ZsMOA5j;5C-6`dx|+hNo>-Q8!UOk7>t=Fd`hyK z-xEqXP&QOXIZYBEH0G-`0J7XlFP(M5468KN8I?u#u5V~7%MWwrQ{b6@;#f%w+kNjU ztw3pqxz;t@5bKynvh7t4g+@Dmfx&Ah4T4ZTh@y~AY9%gVYxK#8#zx_