From 18c97ff38376ecdbd452d1078c53794a7de87a2b Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 6 Sep 2021 16:48:53 +0200 Subject: [PATCH] updating again --- doc/pub/week35/html/week35-bs.html | 29 +++-- doc/pub/week35/html/week35-reveal.html | 83 +++++++++++++- doc/pub/week35/html/week35-solarized.html | 80 ++++++++++++- doc/pub/week35/html/week35.html | 80 ++++++++++++- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 191 bytes doc/pub/week35/ipynb/week35.ipynb | 114 ++++++++++++++++++- doc/src/week35/week35.do.txt | 69 ++++++++++- 7 files changed, 434 insertions(+), 21 deletions(-) diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 10c7c3629..e22a4cc7b 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -230,6 +230,11 @@ Automatically generated HTML file from DocOnce source None, 'rewriting-the-covariance-and-or-correlation-matrix'), ('Linking with the SVD', 2, None, 'linking-with-the-svd'), + ('What does it mean?', 2, None, 'what-does-it-mean'), + ('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$', + 2, + None, + 'and-finally-boldsymbol-x-boldsymbol-x-t'), ('Ridge and LASSO Regression', 2, None, @@ -359,16 +364,18 @@ MathJax.Hub.Config({
  • Correlation Matrix with Pandas and the Franke function
  • Rewriting the Covariance and/or Correlation Matrix
  • Linking with the SVD
    • -
  • Ridge and LASSO Regression
    • -
  • More on Ridge Regression
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • A better understanding of regularization
    • -
  • Decomposing the OLS and Ridge expressions
    • -
  • Mathematical Properties
    • -
  • Exercises for week 36, September 6-10
    • -
  • Exercise 1: Adding Ridge and Lasso Regression
    • -
  •    Exercise: Linear Regression for a two-dimensional function
    • +
  • What does it mean?
    • +
  • And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
    • +
  • Ridge and LASSO Regression
    • +
  • More on Ridge Regression
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • A better understanding of regularization
    • +
  • Decomposing the OLS and Ridge expressions
    • +
  • Mathematical Properties
    • +
  • Exercises for week 36, September 6-10
    • +
  • Exercise 1: Adding Ridge and Lasso Regression
    • +
  •    Exercise: Linear Regression for a two-dimensional function
    • @@ -427,7 +434,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 70
    • +
  • 72
  • »
    • diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 873bd62d3..d42a73f55 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -2726,14 +2726,93 @@ $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$

     
    + + + +

    +

    What does it mean?

    + +

    +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues +given by the singular values squared, that is -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) -with eigenvalues given by the singular values squared, that is

     
    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$

     
    + +

    +In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that +the columns of \( \boldsymbol{V} \) are the eigenvectors of +\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of +\( \boldsymbol{X} \) in a descending order, it means that the column vectors +\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they +encode from the columns of \( \boldsymbol{X} \). + +

    +Note that these are also the eigenvectors and eigenvalues of the +Hessian matrix. + +

    +If we now recall the definition of the covariance matrix (not using +Bessel's correction) we have + +

     
    +$$ +\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, +$$ +

     
    + +

    +meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$, +the number of samples, are the eigenvalues of the covariance +matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square +root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is +self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the +absolute value of the eigenvalues of \( \boldsymbol{X} \). +

    + + +
    +

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    + +

    +For \( \boldsymbol{X}\boldsymbol{X}^T \) we found + +$$ +$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. +

     
    +$$ + +Since the matrices here have dimension \( n\times n \), we have +$$ +

     
    +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, +

     
    +$$ + +leading to +$$ +

     
    +$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. +

     
    +$$ + +

    +Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +$$ +

     
    +$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. +$$ + +

    +It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \). + +

    +Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \).

    diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index bc31b912f..bbcada960 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -250,6 +250,11 @@ div { text-align: justify; text-justify: inter-word; } None, 'rewriting-the-covariance-and-or-correlation-matrix'), ('Linking with the SVD', 2, None, 'linking-with-the-svd'), + ('What does it mean?', 2, None, 'what-does-it-mean'), + ('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$', + 2, + None, + 'and-finally-boldsymbol-x-boldsymbol-x-t'), ('Ridge and LASSO Regression', 2, None, @@ -2718,12 +2723,83 @@ $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) -with eigenvalues given by the singular values squared, that is +

    +









    + +

    What does it mean?

    + +

    +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues +given by the singular values squared, that is + $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ +

    +In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that +the columns of \( \boldsymbol{V} \) are the eigenvectors of +\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of +\( \boldsymbol{X} \) in a descending order, it means that the column vectors +\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they +encode from the columns of \( \boldsymbol{X} \). + +

    +Note that these are also the eigenvectors and eigenvalues of the +Hessian matrix. + +

    +If we now recall the definition of the covariance matrix (not using +Bessel's correction) we have + +$$ +\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, +$$ + +

    +meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$, +the number of samples, are the eigenvalues of the covariance +matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square +root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is +self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the +absolute value of the eigenvalues of \( \boldsymbol{X} \). + +

    +









    + +

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    + +

    +For \( \boldsymbol{X}\boldsymbol{X}^T \) we found + +$$ +$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. +$$ + +Since the matrices here have dimension \( n\times n \), we have +$$ +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, +$$ + +leading to +$$ +$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. +$$ + +

    +Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +$$ +$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. +$$ + +

    +It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \). + +

    +Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +











    diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index b261f20e9..3efd21e35 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -255,6 +255,11 @@ div { text-align: justify; text-justify: inter-word; } None, 'rewriting-the-covariance-and-or-correlation-matrix'), ('Linking with the SVD', 2, None, 'linking-with-the-svd'), + ('What does it mean?', 2, None, 'what-does-it-mean'), + ('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$', + 2, + None, + 'and-finally-boldsymbol-x-boldsymbol-x-t'), ('Ridge and LASSO Regression', 2, None, @@ -2723,12 +2728,83 @@ $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) -with eigenvalues given by the singular values squared, that is +

    +









    + +

    What does it mean?

    + +

    +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues +given by the singular values squared, that is + $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ +

    +In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that +the columns of \( \boldsymbol{V} \) are the eigenvectors of +\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of +\( \boldsymbol{X} \) in a descending order, it means that the column vectors +\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they +encode from the columns of \( \boldsymbol{X} \). + +

    +Note that these are also the eigenvectors and eigenvalues of the +Hessian matrix. + +

    +If we now recall the definition of the covariance matrix (not using +Bessel's correction) we have + +$$ +\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, +$$ + +

    +meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$, +the number of samples, are the eigenvalues of the covariance +matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square +root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is +self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the +absolute value of the eigenvalues of \( \boldsymbol{X} \). + +

    +









    + +

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    + +

    +For \( \boldsymbol{X}\boldsymbol{X}^T \) we found + +$$ +$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. +$$ + +Since the matrices here have dimension \( n\times n \), we have +$$ +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, +$$ + +leading to +$$ +$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. +$$ + +

    +Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +$$ +$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. +$$ + +

    +It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \). + +

    +Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +











    diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 0978c9ee2cc5d88b22010a1cca7cd476b9eecfdc..76b4c6ce14003561849fc4ea3794f866bf04d0e1 100644 GIT binary patch delta 155 zcmV;M0A&Bb0lxtoABzY83@SEZ00ZsMOA5j;5C-6`dx|+hN!p~58!dMjjJZnrlq8{m8LqQvZ&tm4Q*xlVa|LCJo8T+D`{c7?_H%8 zDD5!Ux`rEKo$yGuy~?4`XvZ%wc%Kp00ZsMOA5j;5C-6`dx|+hNt(2g8!dP_jJZmaDM>uP zCzK{Y`A~UGSWJLWmQWS|S#G75&N^X+RhsII%A$JLH?)=IhdJ{p@XSAPtfYnQzIT;Y zptQqW>l$u|HA^Dd_9}-$qaDA%;I)$mL8u-?QAj7X5|^+w`ea07qwv?qc%J8ZUwZ)0 KEx&O92mk