-and for \( \beta_1 \) we have
+and for \( \beta_1 \) we obtain
$$
\beta_1=\frac{2}{1+\lambda},
@@ -786,7 +786,7 @@ which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
-For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \) +For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) and \( beta_1 \). This gives us the following derivatives of the cost function
$$
@@ -796,14 +796,14 @@ $$
$$
-\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
+\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
$$
and
$$
-\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
+\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
$$
@@ -813,11 +813,60 @@ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \
+If we consider the first case, we have then +
+$$
+-4(4-2\beta_0)+\lambda=0,
+$$
+
+
+and
+
+$$
+-2(2-\beta_1)+\lambda=0.
+$$
+
+
+which yields
+
+
+$$
+\beta_0=frac{16+\lambda}{8},
+$$
+
+
+and
+
+$$
+\beta_1=frac{4+\lambda}{2}.
+$$
+
+
+
+Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you. +
+Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. + +
+First we study and compare the OLS and Ridge results. The next code compares all three methods. +
We found above that the outputs \( \boldsymbol{y} \) have a mean value given by \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to -the design matrix are not stocastic variables, we can assume that the +the design matrix are not stochastic variables, we can assume that the probability distribution of our targets is also a normal distribution but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a single output \( y_i \) is given by the Gaussian distribution
$$
-y_i\sim \mathcal{N}(\boldsymbol{X}{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
+y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
$$
-We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution and +We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. +We define this distribution as +
+$$
+p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
+$$
+
+
+which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \).
+
+
+Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have + +
+$$
+p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}).
+$$
+
+More text will be added here.
Friday September 10
+
+
-For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \) +For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) and \( beta_1 \). This gives us the following derivatives of the cost function $$ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), @@ -805,12 +810,12 @@ $$ $$ -\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0, +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, $$ and $$ -\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0. +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. $$ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely @@ -819,9 +824,50 @@ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \
+If we consider the first case, we have then +$$ +-4(4-2\beta_0)+\lambda=0, +$$ + +and +$$ +-2(2-\beta_1)+\lambda=0. +$$ + +which yields + +$$ +\beta_0=frac{16+\lambda}{8}, +$$ + +and +$$ +\beta_1=frac{4+\lambda}{2}. +$$ + +
+Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you. + +
+
+
+
+Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. + +
+First we study and compare the OLS and Ridge results. The next code compares all three methods. + +
We found above that the outputs \( \boldsymbol{y} \) have a mean value given by \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to -the design matrix are not stocastic variables, we can assume that the +the design matrix are not stochastic variables, we can assume that the probability distribution of our targets is also a normal distribution but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a single output \( y_i \) is given by the Gaussian distribution $$ -y_i\sim \mathcal{N}(\boldsymbol{X}{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$
@@ -1028,13 +1074,29 @@ $$
-We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution and +We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. +We define this distribution as +$$ +p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, +$$ + +which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \). + +
+Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have + +$$ +p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}). +$$
+More text will be added here. +
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index 293abf5ef..2a2f4a1dc 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -118,6 +118,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'writing-the-cost-function'),
('Lasso case', 2, None, 'lasso-case'),
+ ('The first Case', 2, None, 'the-first-case'),
+ ('Simple code for solving the above problem',
+ 2,
+ None,
+ 'simple-code-for-solving-the-above-problem'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -783,7 +788,7 @@ $$
\beta_0=\frac{8}{4+\lambda},
$$
-and for \( \beta_1 \) we have
+and for \( \beta_1 \) we obtain
$$
\beta_1=\frac{2}{1+\lambda},
$$
@@ -802,7 +807,7 @@ which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
-For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \) +For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) and \( beta_1 \). This gives us the following derivatives of the cost function $$ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), @@ -810,12 +815,12 @@ $$ $$ -\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0, +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, $$ and $$ -\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0. +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. $$ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely @@ -824,9 +829,50 @@ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \
+If we consider the first case, we have then +$$ +-4(4-2\beta_0)+\lambda=0, +$$ + +and +$$ +-2(2-\beta_1)+\lambda=0. +$$ + +which yields + +$$ +\beta_0=frac{16+\lambda}{8}, +$$ + +and +$$ +\beta_1=frac{4+\lambda}{2}. +$$ + +
+Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you. + +
+
+
+
+Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. + +
+First we study and compare the OLS and Ridge results. The next code compares all three methods. + +
We found above that the outputs \( \boldsymbol{y} \) have a mean value given by \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to -the design matrix are not stocastic variables, we can assume that the +the design matrix are not stochastic variables, we can assume that the probability distribution of our targets is also a normal distribution but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a single output \( y_i \) is given by the Gaussian distribution $$ -y_i\sim \mathcal{N}(\boldsymbol{X}{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$
@@ -1033,13 +1079,29 @@ $$
-We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution and +We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. +We define this distribution as +$$ +p(y_i\vert \boldsymbol{X};\boldsymbol{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, +$$ + +which reads as finding the likelihood of an event \( y_i \) given the input variables \( \boldsymbol{X} \) and the parameters (to be determined) \( \boldsymbol{beta} \). + +
+Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have + +$$ +p(\boldsymbol{y}\vert \boldsymbol{X};\boldsymbol{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \boldsymbol{X};\boldsymbol{beta}). +$$
+More text will be added here. +
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index c989468ec..7b7808b52 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index 920d58234..034c73395 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -933,7 +933,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "and for $\\beta_1$ we have"
+ "and for $\\beta_1$ we obtain"
]
},
{
@@ -969,7 +969,7 @@
"\n",
"## Lasso case\n",
"\n",
- "For Lasso we need now, keeping the same constraint on $\\beta_0^2+\\beta_1^2=1$, to take the derivative of the absolute values of $\\beta_0$\n",
+ "For Lasso we need now, keeping a constraint on $\\vert\\beta_0\\vert+\\vert\\beta_1\\vert=1$, to take the derivative of the absolute values of $\\beta_0$\n",
"and $beta_1$. This gives us the following derivatives of the cost function"
]
},
@@ -987,7 +987,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-2(4-2\\beta_0)+\\lambda\\mathrm{sgn}\\beta_0=0,\n",
+ "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n",
"$$"
]
},
@@ -1003,7 +1003,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}\\beta_1=0.\n",
+ "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n",
"$$"
]
},
@@ -1018,7 +1018,82 @@
"\n",
"3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n",
"\n",
- "4. $\\beta_0 < 0$ and $\\beta_1 < 0$,\n",
+ "4. $\\beta_0 < 0$ and $\\beta_1 < 0$.\n",
+ "\n",
+ "## The first Case\n",
+ "\n",
+ "If we consider the first case, we have then"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "-4(4-2\\beta_0)+\\lambda=0,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "-2(2-\\beta_1)+\\lambda=0.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which yields"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\beta_0=frac{16+\\lambda}{8},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\beta_1=frac{4+\\lambda}{2}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n",
+ "\n",
+ "## Simple code for solving the above problem\n",
+ "\n",
+ "Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\\lambda$, meaning that we need to perform a search in order to find the optimal values.\n",
+ "\n",
+ "First we study and compare the OLS and Ridge results. The next code compares all three methods.\n",
+ "\n",
"\n",
"\n",
"## Linking the regression analysis with a statistical interpretation\n",
@@ -1295,7 +1370,7 @@
"\n",
"We found above that the outputs $\\boldsymbol{y}$ have a mean value given by\n",
"$\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$ and variance $\\sigma^2$. Since the entries to\n",
- "the design matrix are not stocastic variables, we can assume that the\n",
+ "the design matrix are not stochastic variables, we can assume that the\n",
"probability distribution of our targets is also a normal distribution\n",
"but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$. This means that a\n",
"single output $y_i$ is given by the Gaussian distribution"
@@ -1306,7 +1381,7 @@
"metadata": {},
"source": [
"$$\n",
- "y_i\\sim \\mathcal{N}(\\boldsymbol{X}{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
+ "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
"$$"
]
},
@@ -1316,12 +1391,44 @@
"source": [
"## Independent and Identically Distrubuted (iid)\n",
"\n",
- "We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution and \n",
- "\n",
+ "We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution. \n",
+ "We define this distribution as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which reads as finding the likelihood of an event $y_i$ given the input variables $\\boldsymbol{X}$ and the parameters (to be determined) $\\boldsymbol{beta}$.\n",
"\n",
+ "Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\\boldsymbol{y}$ as the product of the single events, that is we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(\\boldsymbol{y}\\vert \\boldsymbol{X};\\boldsymbol{beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i\\vert \\boldsymbol{X};\\boldsymbol{beta}).\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
"## Friday September 10\n",
"\n",
- "\n",
+ "More text will be added here.\n",
"\n",
"## Why resampling methods\n",
"\n",
diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt
index 35f0d253d..b8ef76fd1 100644
--- a/doc/src/week36/week36.do.txt
+++ b/doc/src/week36/week36.do.txt
@@ -512,7 +512,7 @@ and taking the derivative with respect to $\beta_0$ we get
\beta_0=\frac{8}{4+\lambda},
\]
!et
-and for $\beta_1$ we have
+and for $\beta_1$ we obtain
!bt
\[
\beta_1=\frac{2}{1+\lambda},
@@ -530,7 +530,7 @@ which gives $\lambda=4.571$ and $\beta_0=0.933$ and $\beta_1=0.359$.
!split
===== Lasso case =====
-For Lasso we need now, keeping the same constraint on $\beta_0^2+\beta_1^2=1$, to take the derivative of the absolute values of $\beta_0$
+For Lasso we need now, keeping a constraint on $\vert\beta_0\vert+\vert\beta_1\vert=1$, to take the derivative of the absolute values of $\beta_0$
and $beta_1$. This gives us the following derivatives of the cost function
!bt
\[
@@ -540,22 +540,58 @@ C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_
!bt
\[
-\frac{\partialC(\bm{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
+\frac{\partial C(\bm{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
\]
!et
and
!bt
\[
-\frac{\partialC(\bm{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
+\frac{\partial C(\bm{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
\]
!et
We have now four cases to solve besides the trivial cases $\beta_0$ and/or $\beta_1$ are zero, namely
o $\beta_0 > 0$ and $\beta_1 > 0$,
o $\beta_0 > 0$ and $\beta_1 < 0$,
o $\beta_0 < 0$ and $\beta_1 > 0$,
-o $\beta_0 < 0$ and $\beta_1 < 0$,
+o $\beta_0 < 0$ and $\beta_1 < 0$.
+!split
+===== The first Case =====
+If we consider the first case, we have then
+!bt
+\[
+-4(4-2\beta_0)+\lambda=0,
+\]
+!et
+and
+!bt
+\[
+-2(2-\beta_1)+\lambda=0.
+\]
+!et
+which yields
+
+!bt
+\[
+\beta_0=frac{16+\lambda}{8},
+\]
+!et
+and
+!bt
+\[
+\beta_1=frac{4+\lambda}{2}.
+\]
+!et
+
+Using the constraint on $\beta_0$ and $\beta_1$ we can then find the optimal value of $\lambda$ for the different cases. We leave this as an exercise to you.
+
+!split
+===== Simple code for solving the above problem =====
+
+Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\lambda$, meaning that we need to perform a search in order to find the optimal values.
+
+First we study and compare the OLS and Ridge results. The next code compares all three methods.
!split
@@ -734,27 +770,43 @@ $\sigma^2$.
We found above that the outputs $\bm{y}$ have a mean value given by
$\bm{X}\hat{\bm{\beta}}$ and variance $\sigma^2$. Since the entries to
-the design matrix are not stocastic variables, we can assume that the
+the design matrix are not stochastic variables, we can assume that the
probability distribution of our targets is also a normal distribution
but now with mean value $\bm{X}\hat{\bm{\beta}}$. This means that a
single output $y_i$ is given by the Gaussian distribution
!bt
\[
-y_i\sim \mathcal{N}(\bm{X}{i,*}\bm{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
+y_i\sim \mathcal{N}(\bm{X}_{i,*}\bm{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
\]
!et
!split
===== Independent and Identically Distrubuted (iid) =====
-We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution and
+We assume now that the various $y_i$ values are stochastically distributed according to the above Gaussian distribution.
+We define this distribution as
+!bt
+\[
+p(y_i\vert \bm{X};\bm{beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
+\]
+!et
+which reads as finding the likelihood of an event $y_i$ given the input variables $\bm{X}$ and the parameters (to be determined) $\bm{beta}$.
+
+Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\bm{y}$ as the product of the single events, that is we have
+
+!bt
+\[
+p(\bm{y}\vert \bm{X};\bm{beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i\vert \bm{X};\bm{beta}).
+\]
+!et
+
!split
===== Friday September 10 =====
-
+More text will be added here.
!split
===== Why resampling methods =====