From a4d53ac23023445df8b0986863b296e3d00ce99b Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Fri, 10 Sep 2021 08:02:10 +0200 Subject: [PATCH] final Bayes stuff --- doc/pub/week36/html/week36-bs.html | 60 ++++---- doc/pub/week36/html/week36-reveal.html | 96 +++++++++++- doc/pub/week36/html/week36-solarized.html | 82 +++++++++- doc/pub/week36/html/week36.html | 82 +++++++++- doc/pub/week36/ipynb/ipynb-week36-src.tar.gz | Bin 191 -> 192 bytes doc/pub/week36/ipynb/week36.ipynb | 148 ++++++++++++++++++- doc/src/week36/week36.do.txt | 84 ++++++++++- 7 files changed, 514 insertions(+), 38 deletions(-) diff --git a/doc/pub/week36/html/week36-bs.html b/doc/pub/week36/html/week36-bs.html index 060362e3e..b7fdad551 100644 --- a/doc/pub/week36/html/week36-bs.html +++ b/doc/pub/week36/html/week36-bs.html @@ -164,6 +164,8 @@ Automatically generated HTML file from DocOnce source None, 'test-function-for-what-happens-with-ols-ridge-and-lasso'), ("Invoking Bayes' theorem", 2, None, 'invoking-bayes-theorem'), + ('Ridge and Bayes', 2, None, 'ridge-and-bayes'), + ('Lasso and Bayes', 2, None, 'lasso-and-bayes'), ('Why resampling methods', 2, None, 'why-resampling-methods'), ('Resampling methods', 2, None, 'resampling-methods'), ('Resampling approaches can be computationally expensive', @@ -335,33 +337,35 @@ MathJax.Hub.Config({
  • Bayes' Theorem and Ridge and Lasso Regression
  • Test Function for what happens with OLS, Ridge and Lasso
  • Invoking Bayes' theorem
    • -
  • Why resampling methods
    • -
  • Resampling methods
    • -
  • Resampling approaches can be computationally expensive
    • -
  • Why resampling methods ?
    • -
  • Statistical analysis
    • -
  • Resampling methods
    • -
  • Resampling methods: Jackknife and Bootstrap
    • -
  • Resampling methods: Jackknife
    • -
  • Jackknife code example
    • -
  • Resampling methods: Bootstrap
    • -
  • Resampling methods: Bootstrap background
    • -
  • Resampling methods: More Bootstrap background
    • -
  • Resampling methods: Bootstrap approach
    • -
  • Resampling methods: Bootstrap steps
    • -
  • Code example for the Bootstrap method
    • -
  • Various steps in cross-validation
    • -
  • How to set up the cross-validation for Ridge and/or Lasso
    • -
  • Cross-validation in brief
    • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • -
  • The bias-variance tradeoff
    • -
  • Example code for Bias-Variance tradeoff
    • -
  • Understanding what happens
    • -
  • Summing up
    • -
  • Another Example from Scikit-Learn's Repository
    • -
  • More examples on bootstrap and cross-validation and errors
    • -
  • The same example but now with cross-validation
    • -
  • Cross-validation with Ridge
    • +
  • Ridge and Bayes
    • +
  • Lasso and Bayes
    • +
  • Why resampling methods
    • +
  • Resampling methods
    • +
  • Resampling approaches can be computationally expensive
    • +
  • Why resampling methods ?
    • +
  • Statistical analysis
    • +
  • Resampling methods
    • +
  • Resampling methods: Jackknife and Bootstrap
    • +
  • Resampling methods: Jackknife
    • +
  • Jackknife code example
    • +
  • Resampling methods: Bootstrap
    • +
  • Resampling methods: Bootstrap background
    • +
  • Resampling methods: More Bootstrap background
    • +
  • Resampling methods: Bootstrap approach
    • +
  • Resampling methods: Bootstrap steps
    • +
  • Code example for the Bootstrap method
    • +
  • Various steps in cross-validation
    • +
  • How to set up the cross-validation for Ridge and/or Lasso
    • +
  • Cross-validation in brief
    • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • +
  • The bias-variance tradeoff
    • +
  • Example code for Bias-Variance tradeoff
    • +
  • Understanding what happens
    • +
  • Summing up
    • +
  • Another Example from Scikit-Learn's Repository
    • +
  • More examples on bootstrap and cross-validation and errors
    • +
  • The same example but now with cross-validation
    • +
  • Cross-validation with Ridge
    • @@ -420,7 +424,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 74
    • +
  • 76
  • »
    • diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html index ca65ed3c9..3e9398634 100644 --- a/doc/pub/week36/html/week36-reveal.html +++ b/doc/pub/week36/html/week36-reveal.html @@ -1460,7 +1460,7 @@ In the more general case the various inputs should be replaced by the possible f We can now rewrite the above probability as

     
    $$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.. +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$

     
    @@ -1831,7 +1831,7 @@ $$ is given by

     
    $$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.. +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$

     
    @@ -1857,6 +1857,98 @@ We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one +

    +

    Ridge and Bayes

    + +

    +With the posterior probability defined by a likelihood which we have +already modeled and an unknown prior, we are now ready to make +additional models for the prior. + +

    +We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is + +

     
    +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +$$ +

     
    + +

    +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +

     
    +$$ +p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +$$ +

     
    + +

    +We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we +did for OLS, this is most conveniently done by taking the negative +logarithm of the posterior probability. Doing so and leaving out the +constants terms that do not depend on \( \beta \), we have + +

     
    +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ +

     
    + +and replacing \( 1/2\tau^2 \) with \( \lambda \) we have + +

     
    +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ +

     
    + +which is our Ridge cost function! Nice, isn't it? +

    + + +
    +

    Lasso and Bayes

    + +

    +To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is + +

     
    +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +$$ +

     
    + +

    +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +

     
    +$$ +p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +$$ +

     
    + +

    +Taking the negative +logarithm of the posterior probability and leaving out the +constants terms that do not depend on \( \beta \), we have + +

     
    +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ +

     
    + +and replacing \( 1/\tau \) with \( \lambda \) we have + +

     
    +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ +

     
    + +which is our Lasso cost function! +

    + +

    Why resampling methods

    diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html index 6300150c3..f024c75c0 100644 --- a/doc/pub/week36/html/week36-solarized.html +++ b/doc/pub/week36/html/week36-solarized.html @@ -184,6 +184,8 @@ div { text-align: justify; text-justify: inter-word; } None, 'test-function-for-what-happens-with-ols-ridge-and-lasso'), ("Invoking Bayes' theorem", 2, None, 'invoking-bayes-theorem'), + ('Ridge and Bayes', 2, None, 'ridge-and-bayes'), + ('Lasso and Bayes', 2, None, 'lasso-and-bayes'), ('Why resampling methods', 2, None, 'why-resampling-methods'), ('Resampling methods', 2, None, 'resampling-methods'), ('Resampling approaches can be computationally expensive', @@ -1470,7 +1472,7 @@ $$ In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). We can now rewrite the above probability as $$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.. +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$

    @@ -1817,7 +1819,7 @@ $$ is given by $$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.. +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$

    @@ -1839,6 +1841,82 @@ We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one











    +

    Ridge and Bayes

    + +

    +With the posterior probability defined by a likelihood which we have +already modeled and an unknown prior, we are now ready to make +additional models for the prior. + +

    +We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is + +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +$$ + +

    +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +$$ +p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +$$ + +

    +We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we +did for OLS, this is most conveniently done by taking the negative +logarithm of the posterior probability. Doing so and leaving out the +constants terms that do not depend on \( \beta \), we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +and replacing \( 1/2\tau^2 \) with \( \lambda \) we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +which is our Ridge cost function! Nice, isn't it? + +

    +









    + +

    Lasso and Bayes

    + +

    +To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is + +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +$$ + +

    +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +$$ +p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +$$ + +

    +Taking the negative +logarithm of the posterior probability and leaving out the +constants terms that do not depend on \( \beta \), we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ + +and replacing \( 1/\tau \) with \( \lambda \) we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ + +which is our Lasso cost function! + +

    +









    +

    Why resampling methods

    diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index 938b3e4f8..216bca650 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -189,6 +189,8 @@ div { text-align: justify; text-justify: inter-word; } None, 'test-function-for-what-happens-with-ols-ridge-and-lasso'), ("Invoking Bayes' theorem", 2, None, 'invoking-bayes-theorem'), + ('Ridge and Bayes', 2, None, 'ridge-and-bayes'), + ('Lasso and Bayes', 2, None, 'lasso-and-bayes'), ('Why resampling methods', 2, None, 'why-resampling-methods'), ('Resampling methods', 2, None, 'resampling-methods'), ('Resampling approaches can be computationally expensive', @@ -1475,7 +1477,7 @@ $$ In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). We can now rewrite the above probability as $$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.. +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$

    @@ -1822,7 +1824,7 @@ $$ is given by $$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.. +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$

    @@ -1844,6 +1846,82 @@ We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one











    +

    Ridge and Bayes

    + +

    +With the posterior probability defined by a likelihood which we have +already modeled and an unknown prior, we are now ready to make +additional models for the prior. + +

    +We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is + +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +$$ + +

    +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +$$ +p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +$$ + +

    +We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we +did for OLS, this is most conveniently done by taking the negative +logarithm of the posterior probability. Doing so and leaving out the +constants terms that do not depend on \( \beta \), we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +and replacing \( 1/2\tau^2 \) with \( \lambda \) we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +which is our Ridge cost function! Nice, isn't it? + +

    +









    + +

    Lasso and Bayes

    + +

    +To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is + +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +$$ + +

    +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +$$ +p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +$$ + +

    +Taking the negative +logarithm of the posterior probability and leaving out the +constants terms that do not depend on \( \beta \), we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ + +and replacing \( 1/\tau \) with \( \lambda \) we have + +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ + +which is our Lasso cost function! + +

    +









    +

    Why resampling methods

    diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index dfbcacf41bdee588a7394419aeaaa5b842873864..74c7418d7ced8c77e17772cf43ce9427c6dd50e3 100644 GIT binary patch literal 192 zcmV;x06+g9iwFSL^g3Yx1MSaC3c@fD2H>uHia9~aqe<6-E?fvAULd8ZjkQTlQna_X z573q3rihSl^E1pa%p5kW^=^~+yN?!w5aMYHV`iL9iO5_}FlK;Lj5*5)B^(f@jFJe@ zawolX&I>!7(o|=uoz(BsPB)ipOxmhLt=A*A+jA53+kaHAcve{IRI2XV(&QTzu;J|n)QjCCo z@1&>Jd1?Do>heU|k$yGTj~xzg_Drw9Bmcy{R1VhL&X-1mHlA{0D!39=DJH7vbPk0f tH?+XgrB#+d>JDU4D6fo`=eV)vXl3n^_}9+_K@bEVy8{!Bn-Bm9004#mTQ2|r diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 393a2be2b..296edc16b 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -1848,7 +1848,7 @@ "metadata": {}, "source": [ "$$\n", - "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}..\n", + "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", "$$" ] }, @@ -2298,7 +2298,7 @@ "metadata": {}, "source": [ "$$\n", - "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}..\n", + "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", "$$" ] }, @@ -2341,6 +2341,150 @@ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$! \n", "\n", "\n", + "## Ridge and Bayes\n", + "\n", + "With the posterior probability defined by a likelihood which we have\n", + "already modeled and an unknown prior, we are now ready to make\n", + "additional models for the prior.\n", + "\n", + "We can, based on our discussions of the variance of $\\boldsymbol{\\beta}$ and the mean value, assume that the prior for the values $\\boldsymbol{\\beta}$ is given by a Gaussian with mean value zero and variance $\\tau^2$, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\beta_j^2}{2\\tau^2}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p(\\boldsymbol{\\beta}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\beta_j^2}{2\\tau^2}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", + "did for OLS, this is most conveniently done by taking the negative\n", + "logarithm of the posterior probability. Doing so and leaving out the\n", + "constants terms that do not depend on $\\beta$, we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and replacing $1/2\\tau^2$ with $\\lambda$ we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which is our Ridge cost function! Nice, isn't it?\n", + "\n", + "## Lasso and Bayes\n", + "\n", + "To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution ([Laplace in this case](https://en.wikipedia.org/wiki/Laplace_distribution)) with zero mean value, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\vert\\beta_j\\vert}{\\tau}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p(\\boldsymbol{\\beta}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\vert\\beta_j\\vert}{\\tau}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Taking the negative\n", + "logarithm of the posterior probability and leaving out the\n", + "constants terms that do not depend on $\\beta$, we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and replacing $1/\\tau$ with $\\lambda$ we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which is our Lasso cost function! \n", "\n", "\n", "\n", diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index 1227fb798..cb02c6fdf 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -1127,7 +1127,7 @@ In the more general case the various inputs should be replaced by the possible f We can now rewrite the above probability as !bt \[ -p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.. +p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}. \] !et @@ -1441,7 +1441,7 @@ For ordinary least squares we postulated that the maximum likelihood for the doa is given by !bt \[ -p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.. +p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}. \] !et @@ -1463,6 +1463,86 @@ p(\bm{\beta}\vert\bm{D})\propto p(\bm{D}\vert\bm{\beta})p(\bm{\beta}). We have a model for $p(\bm{D}\vert\bm{\beta})$ but need one for the _prior_ $p(\bm{\beta}$! +!split +===== Ridge and Bayes ===== + +With the posterior probability defined by a likelihood which we have +already modeled and an unknown prior, we are now ready to make +additional models for the prior. + +We can, based on our discussions of the variance of $\bm{\beta}$ and the mean value, assume that the prior for the values $\bm{\beta}$ is given by a Gaussian with mean value zero and variance $\tau^2$, that is + +!bt +\[ +p(\bm{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +\] +!et + +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +!bt +\[ +p(\bm{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}. +\] +!et + + +We can now optimize this quantity with respect to $\bm{\beta}$. As we +did for OLS, this is most conveniently done by taking the negative +logarithm of the posterior probability. Doing so and leaving out the +constants terms that do not depend on $\beta$, we have + + +!bt +\[ +C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\bm{\beta}\vert\vert_2^2, +\] +!et +and replacing $1/2\tau^2$ with $\lambda$ we have + +!bt +\[ +C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_2^2, +\] +!et +which is our Ridge cost function! Nice, isn't it? + +!split +===== Lasso and Bayes ===== + +To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution ("Laplace in this case":"https://en.wikipedia.org/wiki/Laplace_distribution") with zero mean value, that is + +!bt +\[ +p(\bm{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +\] +!et + +Our posterior probability becomes then (omitting the normalization factor which is just a constant) +!bt +\[ +p(\bm{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}. +\] +!et + + +Taking the negative +logarithm of the posterior probability and leaving out the +constants terms that do not depend on $\beta$, we have + + +!bt +\[ +C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\bm{\beta}\vert\vert_1, +\] +!et +and replacing $1/\tau$ with $\lambda$ we have + +!bt +\[ +C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_1, +\] +!et +which is our Lasso cost function!