From 9c3168fd8359a221e25d299f892fc3199a195517 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Wed, 8 Sep 2021 06:32:38 +0200 Subject: [PATCH] correcting typos --- doc/pub/week36/html/week36-bs.html | 80 ++++++++++--------- doc/pub/week36/html/week36-reveal.html | 13 ++- doc/pub/week36/html/week36-solarized.html | 14 +++- doc/pub/week36/html/week36.html | 14 +++- doc/pub/week36/ipynb/ipynb-week36-src.tar.gz | Bin 192 -> 191 bytes doc/pub/week36/ipynb/week36.ipynb | 10 ++- doc/src/week36/week36.do.txt | 11 ++- 7 files changed, 83 insertions(+), 59 deletions(-) diff --git a/doc/pub/week36/html/week36-bs.html b/doc/pub/week36/html/week36-bs.html index 3c356d49c..61a1699bc 100644 --- a/doc/pub/week36/html/week36-bs.html +++ b/doc/pub/week36/html/week36-bs.html @@ -68,6 +68,7 @@ Automatically generated HTML file from DocOnce source 2, None, 'deriving-the-ridge-regression-equations'), + ('SVD analysis', 2, None, 'svd-analysis'), ('Interpreting the Ridge results', 2, None, @@ -233,44 +234,45 @@ MathJax.Hub.Config({
  • Ridge and LASSO Regression
  • From OLS to Ridge and Lasso
  • Deriving the Ridge Regression Equations
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • Deriving the Lasso Regression Equations
    • -
  • Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
    • -
  • Ridge Regression
    • -
  • Lasso Regression
    • -
  • Linking the regression analysis with a statistical interpretation
    • -
  • Assumptions made
    • -
  • Expectation value and variance
    • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • -
  • Friday September 10
    • -
  • Why resampling methods
    • -
  • Resampling methods
    • -
  • Resampling approaches can be computationally expensive
    • -
  • Why resampling methods ?
    • -
  • Statistical analysis
    • -
  • Resampling methods
    • -
  • Resampling methods: Jackknife and Bootstrap
    • -
  • Resampling methods: Jackknife
    • -
  • Jackknife code example
    • -
  • Resampling methods: Bootstrap
    • -
  • Resampling methods: Bootstrap background
    • -
  • Resampling methods: More Bootstrap background
    • -
  • Resampling methods: Bootstrap approach
    • -
  • Resampling methods: Bootstrap steps
    • -
  • Code example for the Bootstrap method
    • -
  • Various steps in cross-validation
    • -
  • How to set up the cross-validation for Ridge and/or Lasso
    • -
  • Cross-validation in brief
    • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • -
  • The bias-variance tradeoff
    • -
  • Example code for Bias-Variance tradeoff
    • -
  • Understanding what happens
    • -
  • Summing up
    • -
  • Another Example from Scikit-Learn's Repository
    • -
  • More examples on bootstrap and cross-validation and errors
    • -
  • The same example but now with cross-validation
    • -
  • Cross-validation with Ridge
    • +
  • SVD analysis
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • Deriving the Lasso Regression Equations
    • +
  • Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
    • +
  • Ridge Regression
    • +
  • Lasso Regression
    • +
  • Linking the regression analysis with a statistical interpretation
    • +
  • Assumptions made
    • +
  • Expectation value and variance
    • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • +
  • Friday September 10
    • +
  • Why resampling methods
    • +
  • Resampling methods
    • +
  • Resampling approaches can be computationally expensive
    • +
  • Why resampling methods ?
    • +
  • Statistical analysis
    • +
  • Resampling methods
    • +
  • Resampling methods: Jackknife and Bootstrap
    • +
  • Resampling methods: Jackknife
    • +
  • Jackknife code example
    • +
  • Resampling methods: Bootstrap
    • +
  • Resampling methods: Bootstrap background
    • +
  • Resampling methods: More Bootstrap background
    • +
  • Resampling methods: Bootstrap approach
    • +
  • Resampling methods: Bootstrap steps
    • +
  • Code example for the Bootstrap method
    • +
  • Various steps in cross-validation
    • +
  • How to set up the cross-validation for Ridge and/or Lasso
    • +
  • Cross-validation in brief
    • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • +
  • The bias-variance tradeoff
    • +
  • Example code for Bias-Variance tradeoff
    • +
  • Understanding what happens
    • +
  • Summing up
    • +
  • Another Example from Scikit-Learn's Repository
    • +
  • More examples on bootstrap and cross-validation and errors
    • +
  • The same example but now with cross-validation
    • +
  • Cross-validation with Ridge
    • @@ -329,7 +331,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 48
    • +
  • 49
  • »
    • diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html index 2451a7d21..45bc94c05 100644 --- a/doc/pub/week36/html/week36-reveal.html +++ b/doc/pub/week36/html/week36-reveal.html @@ -189,7 +189,7 @@ $$ $$

     
    -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix

     
    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, @@ -447,6 +447,11 @@ modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The conseq particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. + + + +

    +

    SVD analysis

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) @@ -569,7 +574,7 @@ $$ and reordering we have

     
    $$ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. $$

     
    @@ -585,7 +590,7 @@ Let us assume that our design matrix is given by unit (identity) matrix, that is diagonal. In this case we have an equal number of rows and columns \( n=p \).

    -Our model approximation is just \( \tild{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) +Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))

     
    $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, @@ -644,7 +649,7 @@ which leads to $$ \hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ - 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. $$

     
    diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html index 1e86b8369..f60448f4d 100644 --- a/doc/pub/week36/html/week36-solarized.html +++ b/doc/pub/week36/html/week36-solarized.html @@ -88,6 +88,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'deriving-the-ridge-regression-equations'), + ('SVD analysis', 2, None, 'svd-analysis'), ('Interpreting the Ridge results', 2, None, @@ -280,7 +281,7 @@ $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, $$ @@ -497,6 +498,11 @@ particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    +









    + +

    SVD analysis

    +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as @@ -599,7 +605,7 @@ $$ and reordering we have $$ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. $$ This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. @@ -614,7 +620,7 @@ Let us assume that our design matrix is given by unit (identity) matrix, that is diagonal. In this case we have an equal number of rows and columns \( n=p \).

    -Our model approximation is just \( \tild{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) +Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, $$ @@ -660,7 +666,7 @@ which leads to $$ \hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ - 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. $$

    diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index 5d9cdcfc9..b129c6975 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -93,6 +93,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'deriving-the-ridge-regression-equations'), + ('SVD analysis', 2, None, 'svd-analysis'), ('Interpreting the Ridge results', 2, None, @@ -285,7 +286,7 @@ $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, $$ @@ -502,6 +503,11 @@ particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    +









    + +

    SVD analysis

    +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as @@ -604,7 +610,7 @@ $$ and reordering we have $$ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. $$ This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. @@ -619,7 +625,7 @@ Let us assume that our design matrix is given by unit (identity) matrix, that is diagonal. In this case we have an equal number of rows and columns \( n=p \).

    -Our model approximation is just \( \tild{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) +Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, $$ @@ -665,7 +671,7 @@ which leads to $$ \hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ - 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. $$

    diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index 1413466271d342cb2ecbad013ab60078aafae093..ed6bbb6e807183418c81b200a35be34a100e7b82 100644 GIT binary patch delta 169 zcmV;a09OCN0lxtyABzY8*gQC400ZsM%?iRW3T!m delta 170 zcmV;b09F6L0l)zzABzY8Ha9q700ZsM%?iRW3 \\frac{\\lambda}{2}\\\\\n", " y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n", - "\t\t\t\t\t\t\t 0 &\\mathrm{if} & \\verty_i\\vert\\le \\frac{\\lambda}{2}\\\\\n", + "\t\t\t\t\t\t\t 0 &\\mathrm{if} & \\verty_i\\vert\\le \\frac{\\lambda}{2}\\end{array}\\right.\\\\.\n", "$$" ] }, diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index 4299321aa..826c4575f 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -26,7 +26,7 @@ We used the SVD to analyse the matrix to invert in ordinary lineat regression \bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T. \] !et -Since the matrices here have dimension $p\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix +Since the matrices here have dimension $p\times p$, with $p$ corresponding to the singular values, we defined last week the matrix !bt \[ \bm{\Sigma}^T\bm{\Sigma} = \begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix}, @@ -251,6 +251,9 @@ particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of $\lambda$, we may even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +!split +===== SVD analysis ===== + Using our insights about the SVD of the design matrix $\bm{X}$ We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as !bt @@ -352,7 +355,7 @@ we have that the derivative of the cost function is and reordering we have !bt \[ -\bm{X}^T\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=2\bm{X}^T(\bm{y}. +\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}. \] !et This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later. @@ -366,7 +369,7 @@ This equation does not lead to a nice analytical equation as in Ridge regression Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the diagonal. In this case we have an equal number of rows and columns $n=p$. -Our model approximation is just $\tild{\bm{y}}=\bm{\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$) +Our model approximation is just $\tilde{\bm{y}}=\bm{\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$) !bt \[ C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, @@ -416,7 +419,7 @@ which leads to \[ \hat{\bm{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\ - 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. \] !et