From 81e322f5109e1071614ec26fcc40f8ebe55b2084 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Sun, 5 Sep 2021 22:42:30 +0200 Subject: [PATCH] updating week 35 --- doc/pub/week35/html/week35-bs.html | 41 ++++---- doc/pub/week35/html/week35-reveal.html | 53 ++++++++++ doc/pub/week35/html/week35-solarized.html | 45 +++++++++ doc/pub/week35/html/week35.html | 45 +++++++++ doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 100 +++++++++++++++++++ doc/src/week35/week35.do.txt | 48 +++++++++ 7 files changed, 314 insertions(+), 18 deletions(-) diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 277c530c3..d73383b95 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -192,6 +192,10 @@ Automatically generated HTML file from DocOnce source 2, None, 'setting-up-the-matrix-to-be-inverted'), + ('Further properties (important gems for our analysis)', + 2, + None, + 'further-properties-important-gems-for-our-analysis'), ('Ridge and LASSO Regression', 2, None, @@ -335,23 +339,24 @@ MathJax.Hub.Config({
  • Matheamtics of the SVD and implications
  • Example Matrix
  • Setting up the Matrix to be inverted
    • -
  • Ridge and LASSO Regression
    • -
  • More on Ridge Regression
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • A better understanding of regularization
    • -
  • Decomposing the OLS and Ridge expressions
    • -
  • Introducing the Covariance and Correlation functions
    • -
  • Correlation Function and Design/Feature Matrix
    • -
  • Covariance Matrix Examples
    • -
  • Correlation Matrix
    • -
  • Correlation Matrix with Pandas
    • -
  • Correlation Matrix with Pandas and the Franke function
    • -
  • Rewriting the Covariance and/or Correlation Matrix
    • -
  • Mathematical Properties
    • -
  • Exercises for week 36, September 6-10
    • -
  • Exercise 1: Adding Ridge and Lasso Regression
    • -
  •    Exercise: Linear Regression for a two-dimensional function
    • +
  • Further properties (important gems for our analysis)
    • +
  • Ridge and LASSO Regression
    • +
  • More on Ridge Regression
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • A better understanding of regularization
    • +
  • Decomposing the OLS and Ridge expressions
    • +
  • Introducing the Covariance and Correlation functions
    • +
  • Correlation Function and Design/Feature Matrix
    • +
  • Covariance Matrix Examples
    • +
  • Correlation Matrix
    • +
  • Correlation Matrix with Pandas
    • +
  • Correlation Matrix with Pandas and the Franke function
    • +
  • Rewriting the Covariance and/or Correlation Matrix
    • +
  • Mathematical Properties
    • +
  • Exercises for week 36, September 6-10
    • +
  • Exercise 1: Adding Ridge and Lasso Regression
    • +
  •    Exercise: Linear Regression for a two-dimensional function
    • @@ -410,7 +415,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 66
    • +
  • 67
  • »
    • diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 5c2d75776..30ffce852 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -2181,6 +2181,59 @@ It means that the ordinary least square model (with the optimal parameters) \( \ +
    +

    Further properties (important gems for our analysis)

    + +

    +Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, +

     
    +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ +

     
    + +

    +If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

     
    +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. +$$ +

     
    + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +

     
    +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ +

     
    + +

    +Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have +

     
    +$$ +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. +$$ +

     
    + +

    +If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get +

     
    +$$ +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. +$$ +

     
    + +This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) +with eigenvalues given by the singular values squared, that is +

     
    +$$ +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. +$$ +

     
    +

    + +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index 3a825637b..27bfce031 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -212,6 +212,10 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'setting-up-the-matrix-to-be-inverted'), + ('Further properties (important gems for our analysis)', + 2, + None, + 'further-properties-important-gems-for-our-analysis'), ('Ridge and LASSO Regression', 2, None, @@ -2221,6 +2225,47 @@ It means that the ordinary least square model (with the optimal parameters) \( \











    +

    Further properties (important gems for our analysis)

    + +

    +Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +

    +If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. +$$ + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ + +

    +Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have +$$ +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. +$$ + +

    +If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get +$$ +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. +$$ + +This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. +$$ + +

    +









    +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index fd3b8ab8f..cc8a3a179 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -217,6 +217,10 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'setting-up-the-matrix-to-be-inverted'), + ('Further properties (important gems for our analysis)', + 2, + None, + 'further-properties-important-gems-for-our-analysis'), ('Ridge and LASSO Regression', 2, None, @@ -2226,6 +2230,47 @@ It means that the ordinary least square model (with the optimal parameters) \( \











    +

    Further properties (important gems for our analysis)

    + +

    +Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +

    +If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. +$$ + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ + +

    +Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have +$$ +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. +$$ + +

    +If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get +$$ +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. +$$ + +This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. +$$ + +

    +









    +

    Ridge and LASSO Regression

    diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 246c38ddc0c4e15a5b538b3c39d82bc11ee722cd..9141f751bccd35ec637d7c0a1388f116458452b2 100644 GIT binary patch delta 157 zcmV;O0Al~Z0l)zpABzY8cq=tw00ZsMOA5j;5C-6`dx|+hNz$f~8!UNglE9c*%BLi3 zeorXnK$D>|#(747(3mqG09kIOm(Dt2hEV*Eh74<%c=*De%lcajc|;?Y?)F zR-m-QT1J^i#f}H&^Tc{0J7XlFP(M5468KN8I?u#u5V~7%MWwrQ{b6@;#f%w+kNjU ztw3pqxz;t@5bHRJWZSD83XOLB0)y918U&$w5Je%K)Jj~!*65QFjg7)zALDtR=Y8z~ LE?2E_00;m8wJ$?G diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 65762ee72..42790a125 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -2743,6 +2743,106 @@ "\n", "It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$.\n", "\n", + "## Further properties (important gems for our analysis)\n", + "\n", + "Let us study again $\\boldsymbol{X}^T\\boldsymbol{X}$ in terms of our SVD," + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If we now multiply from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n", + "with eigenvalues given by the singular values squared, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Similarly, if we use the SVD decomposition for the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$, we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If we now multiply from the right with $\\boldsymbol{U}$ (using the orthogonality of $\\boldsymbol{U}$) we get" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This means the vectors $\\boldsymbol{u}_i$ of the orthogonal matrix $\\boldsymbol{U}$ are the eigenvectors of the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$\n", + "with eigenvalues given by the singular values squared, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{u}_i=\\boldsymbol{u}_i\\sigma_i^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ "## Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index c38b44933..9cee6b018 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -1729,6 +1729,54 @@ that belong to $i>p-1$, give all zeros when we perform the multiplications. This It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$. +!split +===== Further properties (important gems for our analysis) ===== + +Let us study again $\bm{X}^T\bm{X}$ in terms of our SVD, +!bt +\[ +\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T. +\] +!et + +If we now multiply from the right with $\bm{V}$ (using the orthogonality of $\bm{V}$) we get +!bt +\[ +\left(\bm{X}^T\bm{X}\right)\bm{V}=\bm{V}\bm{\Sigma}^T\bm{\Sigma}. +\] +!et +This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$ are the eigenvectors of the matrix $\bm{X}^T\bm{X}$ +with eigenvalues given by the singular values squared, that is +!bt +\[ +\left(\bm{X}^T\bm{X}\right)\bm{v}_i=\bm{v}_i\sigma_i^2. +\] +!et + +Similarly, if we use the SVD decomposition for the matrix $\bm{X}\bm{X}^T$, we have +!bt +\[ +\bm{X}\bm{X}^T=\bm{U}\bm{\Sigma}\bm{V}^T\bm{V}\bm{\Sigma}^T\bm{U}^T=\bm{U}\bm{\Sigma}\bm{\Sigma}^T\bm{U}^T. +\] +!et + +If we now multiply from the right with $\bm{U}$ (using the orthogonality of $\bm{U}$) we get +!bt +\[ +\left(\bm{X}\bm{X}^T\right)\bm{U}=\bm{U}\bm{\Sigma}\bm{\Sigma}^T. +\] +!et +This means the vectors $\bm{u}_i$ of the orthogonal matrix $\bm{U}$ are the eigenvectors of the matrix $\bm{X}\bm{X}^T$ +with eigenvalues given by the singular values squared, that is +!bt +\[ +\left(\bm{X}\bm{X}^T\right)\bm{u}_i=\bm{u}_i\sigma_i^2. +\] +!et + + + + !split ===== Ridge and LASSO Regression =====