$$
-p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}..
+p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
@@ -1831,7 +1831,7 @@ $$
is given by
$$
-p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}..
+p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
+With the posterior probability defined by a likelihood which we have
+already modeled and an unknown prior, we are now ready to make
+additional models for the prior.
+
+
+We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is
+
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+
+We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we
+did for OLS, this is most conveniently done by taking the negative
+logarithm of the posterior probability. Doing so and leaving out the
+constants terms that do not depend on \( \beta \), we have
+
+
+To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is
+
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+
+Taking the negative
+logarithm of the posterior probability and leaving out the
+constants terms that do not depend on \( \beta \), we have
+
+
@@ -1817,7 +1819,7 @@ $$
is given by
$$
-p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}..
+p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
@@ -1839,6 +1841,82 @@ We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one
+With the posterior probability defined by a likelihood which we have
+already modeled and an unknown prior, we are now ready to make
+additional models for the prior.
+
+
+We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is
+
+$$
+p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+$$
+
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+$$
+p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+$$
+
+
+We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we
+did for OLS, this is most conveniently done by taking the negative
+logarithm of the posterior probability. Doing so and leaving out the
+constants terms that do not depend on \( \beta \), we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
+$$
+
+and replacing \( 1/2\tau^2 \) with \( \lambda \) we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
+$$
+
+which is our Ridge cost function! Nice, isn't it?
+
+
+
+To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is
+
+$$
+p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+$$
+
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+$$
+p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+$$
+
+
+Taking the negative
+logarithm of the posterior probability and leaving out the
+constants terms that do not depend on \( \beta \), we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
+$$
+
+and replacing \( 1/\tau \) with \( \lambda \) we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+$$
+
+which is our Lasso cost function!
+
+
+
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index 938b3e4f8..216bca650 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -189,6 +189,8 @@ div { text-align: justify; text-justify: inter-word; }
None,
'test-function-for-what-happens-with-ols-ridge-and-lasso'),
("Invoking Bayes' theorem", 2, None, 'invoking-bayes-theorem'),
+ ('Ridge and Bayes', 2, None, 'ridge-and-bayes'),
+ ('Lasso and Bayes', 2, None, 'lasso-and-bayes'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
@@ -1475,7 +1477,7 @@ $$
In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \).
We can now rewrite the above probability as
$$
-p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}..
+p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
@@ -1822,7 +1824,7 @@ $$
is given by
$$
-p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}..
+p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
@@ -1844,6 +1846,82 @@ We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one
+With the posterior probability defined by a likelihood which we have
+already modeled and an unknown prior, we are now ready to make
+additional models for the prior.
+
+
+We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is
+
+$$
+p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+$$
+
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+$$
+p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+$$
+
+
+We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we
+did for OLS, this is most conveniently done by taking the negative
+logarithm of the posterior probability. Doing so and leaving out the
+constants terms that do not depend on \( \beta \), we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
+$$
+
+and replacing \( 1/2\tau^2 \) with \( \lambda \) we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
+$$
+
+which is our Ridge cost function! Nice, isn't it?
+
+
+
+To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is
+
+$$
+p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+$$
+
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+$$
+p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+$$
+
+
+Taking the negative
+logarithm of the posterior probability and leaving out the
+constants terms that do not depend on \( \beta \), we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
+$$
+
+and replacing \( 1/\tau \) with \( \lambda \) we have
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+$$
+
+which is our Lasso cost function!
+
+
+
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index dfbcacf41..74c7418d7 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index 393a2be2b..296edc16b 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -1848,7 +1848,7 @@
"metadata": {},
"source": [
"$$\n",
- "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}..\n",
+ "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
"$$"
]
},
@@ -2298,7 +2298,7 @@
"metadata": {},
"source": [
"$$\n",
- "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}..\n",
+ "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
"$$"
]
},
@@ -2341,6 +2341,150 @@
"We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$! \n",
"\n",
"\n",
+ "## Ridge and Bayes\n",
+ "\n",
+ "With the posterior probability defined by a likelihood which we have\n",
+ "already modeled and an unknown prior, we are now ready to make\n",
+ "additional models for the prior.\n",
+ "\n",
+ "We can, based on our discussions of the variance of $\\boldsymbol{\\beta}$ and the mean value, assume that the prior for the values $\\boldsymbol{\\beta}$ is given by a Gaussian with mean value zero and variance $\\tau^2$, that is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\beta_j^2}{2\\tau^2}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(\\boldsymbol{\\beta}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\beta_j^2}{2\\tau^2}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n",
+ "did for OLS, this is most conveniently done by taking the negative\n",
+ "logarithm of the posterior probability. Doing so and leaving out the\n",
+ "constants terms that do not depend on $\\beta$, we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and replacing $1/2\\tau^2$ with $\\lambda$ we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which is our Ridge cost function! Nice, isn't it?\n",
+ "\n",
+ "## Lasso and Bayes\n",
+ "\n",
+ "To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution ([Laplace in this case](https://en.wikipedia.org/wiki/Laplace_distribution)) with zero mean value, that is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\vert\\beta_j\\vert}{\\tau}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(\\boldsymbol{\\beta}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{-(\\frac{\\vert\\beta_j\\vert}{\\tau}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Taking the negative\n",
+ "logarithm of the posterior probability and leaving out the\n",
+ "constants terms that do not depend on $\\beta$, we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and replacing $1/\\tau$ with $\\lambda$ we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which is our Lasso cost function! \n",
"\n",
"\n",
"\n",
diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt
index 1227fb798..cb02c6fdf 100644
--- a/doc/src/week36/week36.do.txt
+++ b/doc/src/week36/week36.do.txt
@@ -1127,7 +1127,7 @@ In the more general case the various inputs should be replaced by the possible f
We can now rewrite the above probability as
!bt
\[
-p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}..
+p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
\]
!et
@@ -1441,7 +1441,7 @@ For ordinary least squares we postulated that the maximum likelihood for the doa
is given by
!bt
\[
-p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}..
+p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
\]
!et
@@ -1463,6 +1463,86 @@ p(\bm{\beta}\vert\bm{D})\propto p(\bm{D}\vert\bm{\beta})p(\bm{\beta}).
We have a model for $p(\bm{D}\vert\bm{\beta})$ but need one for the _prior_ $p(\bm{\beta}$!
+!split
+===== Ridge and Bayes =====
+
+With the posterior probability defined by a likelihood which we have
+already modeled and an unknown prior, we are now ready to make
+additional models for the prior.
+
+We can, based on our discussions of the variance of $\bm{\beta}$ and the mean value, assume that the prior for the values $\bm{\beta}$ is given by a Gaussian with mean value zero and variance $\tau^2$, that is
+
+!bt
+\[
+p(\bm{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+\]
+!et
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+!bt
+\[
+p(\bm{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+\]
+!et
+
+
+We can now optimize this quantity with respect to $\bm{\beta}$. As we
+did for OLS, this is most conveniently done by taking the negative
+logarithm of the posterior probability. Doing so and leaving out the
+constants terms that do not depend on $\beta$, we have
+
+
+!bt
+\[
+C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\bm{\beta}\vert\vert_2^2,
+\]
+!et
+and replacing $1/2\tau^2$ with $\lambda$ we have
+
+!bt
+\[
+C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_2^2,
+\]
+!et
+which is our Ridge cost function! Nice, isn't it?
+
+!split
+===== Lasso and Bayes =====
+
+To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution ("Laplace in this case":"https://en.wikipedia.org/wiki/Laplace_distribution") with zero mean value, that is
+
+!bt
+\[
+p(\bm{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+\]
+!et
+
+Our posterior probability becomes then (omitting the normalization factor which is just a constant)
+!bt
+\[
+p(\bm{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+\]
+!et
+
+
+Taking the negative
+logarithm of the posterior probability and leaving out the
+constants terms that do not depend on $\beta$, we have
+
+
+!bt
+\[
+C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\bm{\beta}\vert\vert_1,
+\]
+!et
+and replacing $1/\tau$ with $\lambda$ we have
+
+!bt
+\[
+C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
+\]
+!et
+which is our Lasso cost function!
@@ -1857,6 +1857,98 @@ We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one
+Ridge and Bayes
+
+
+$$
+p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+$$
+
+
+
+$$
+p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\beta_j^2}{2\tau^2}}.
+$$
+
+
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
+$$
+
+
+and replacing \( 1/2\tau^2 \) with \( \lambda \) we have
+
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
+$$
+
+
+which is our Ridge cost function! Nice, isn't it?
+Lasso and Bayes
+
+
+$$
+p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+$$
+
+
+
+$$
+p(\boldsymbol{\beta}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{-(\frac{\vert\beta_j\vert}{\tau}}.
+$$
+
+
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
+$$
+
+
+and replacing \( 1/\tau \) with \( \lambda \) we have
+
+
+$$
+C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+$$
+
+
+which is our Lasso cost function!
+Why resampling methods
diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html
index 6300150c3..f024c75c0 100644
--- a/doc/pub/week36/html/week36-solarized.html
+++ b/doc/pub/week36/html/week36-solarized.html
@@ -184,6 +184,8 @@ div { text-align: justify; text-justify: inter-word; }
None,
'test-function-for-what-happens-with-ols-ridge-and-lasso'),
("Invoking Bayes' theorem", 2, None, 'invoking-bayes-theorem'),
+ ('Ridge and Bayes', 2, None, 'ridge-and-bayes'),
+ ('Lasso and Bayes', 2, None, 'lasso-and-bayes'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
@@ -1470,7 +1472,7 @@ $$
In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \).
We can now rewrite the above probability as
$$
-p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}..
+p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
+Ridge and Bayes
+
+
+
+Lasso and Bayes
+
+
+
Why resampling methods
+Ridge and Bayes
+
+
+
+Lasso and Bayes
+
+
+
Why resampling methods