-There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). -Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). +There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).
To see this, let us write the cost function for Ridge regression. @@ -1173,17 +1173,61 @@ $$ $$
-which leads to
+which leads to the well-known OLS equation for the optimal paramters \( \beta \)
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
+\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
$$
+
+
+Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics.
+A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.
+Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.
+
+
+Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).
+
+
+We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively.
+The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \).
+
+
+
+
+Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
Friday September 10
+More basic Statistics and Bayes' theorem
+
+
+$$
+p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
+$$
+
+
+$$
+p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x),
+$$
+
+
+where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).
+Frliday September 10
+
+
-There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). -Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). +There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).
To see this, let us write the cost function for Ridge regression. @@ -1148,15 +1152,59 @@ $$ \boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, $$ -which leads to +which leads to the well-known OLS equation for the optimal paramters \( \beta \) $$ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! $$ +
+Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. +
-
+A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. + +
+Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). + +
+We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). + +
+
+$$ +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +$$ +
+
+$$ +p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x), +$$ + +where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). +
+
+
+
+Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index 0a067dd6e..d44b27577 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -150,7 +150,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'maximum-likelihood-estimation-mle'),
('A new Cost Function', 2, None, 'a-new-cost-function'),
- ('Friday September 10', 2, None, 'friday-september-10'),
+ ("More basic Statistics and Bayes' theorem",
+ 2,
+ None,
+ 'more-basic-statistics-and-bayes-theorem'),
+ ('Frliday September 10', 2, None, 'frliday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
@@ -766,8 +770,8 @@ $$
$$
-There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). -Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). +There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).
To see this, let us write the cost function for Ridge regression. @@ -1153,15 +1157,59 @@ $$ \boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, $$ -which leads to +which leads to the well-known OLS equation for the optimal paramters \( \beta \) $$ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! $$ +
+Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. +
-
+A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. + +
+Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). + +
+We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). + +
+
+$$ +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +$$ +
+
+$$ +p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x), +$$ + +where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). +
+
+
+
+Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index 45b8b91ed..89a5af289 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index 3f50ff006..bdb0e54f0 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -884,8 +884,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2 via the parameter $\\lambda$.\n",
- "Let us for simplicity assume now that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\\beta$ and $\\lambda$.\n",
+ "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n",
+ "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n",
"\n",
"To see this, let us write the cost function for Ridge regression. \n",
"\n",
@@ -1503,7 +1503,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "which leads to"
+ "which leads to the well-known OLS equation for the optimal paramters $\\beta$"
]
},
{
@@ -1511,7 +1511,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
+ "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
"$$"
]
},
@@ -1519,8 +1519,59 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Friday September 10\n",
+ "Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. \n",
"\n",
+ "## More basic Statistics and Bayes' theorem\n",
+ "\n",
+ "A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.\n",
+ "Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.\n",
+ "\n",
+ "Assume we have two domains of events $X=[x_0,x_1,\\dots,x_{n-1}]$ and $Y=[y_0,y_1,\\dots,y_{n-1}]$.\n",
+ "\n",
+ "We define also the likelihood for $X$ and $Y$ as $p(X)$ and $p(Y)$ respectively.\n",
+ "The likelihood of a specific event $x_i$ (or $y_i$) is then written as $p(X=x_i)$ or just $p(x_i)=p_i$. \n",
+ "\n",
+ "**Union of events is given by.**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**The product rule is given by.**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "p(X \\cup Y)= p(X,y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(x),\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Frliday September 10\n",
+ "\n",
+ "Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.\n",
"\n",
"\n",
"## Why resampling methods\n",
diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt
index 047fe5215..d5a3d8dc7 100644
--- a/doc/src/week36/week36.do.txt
+++ b/doc/src/week36/week36.do.txt
@@ -485,8 +485,8 @@ Inserting the above values we obtain that
\]
!et
-There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2 via the parameter $\lambda$.
-Let us for simplicity assume now that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\beta$ and $\lambda$.
+There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2$ via the parameter $\lambda$.
+Let us for simplicity assume that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\beta$ and $\lambda$.
To see this, let us write the cost function for Ridge regression.
@@ -853,17 +853,51 @@ Taking the derivative of the *new* cost function with respect to the parameters
\bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right) =0,
\]
!et
-which leads to
+which leads to the well-known OLS equation for the optimal paramters $\beta$
!bt
\[
-\hat{\bm{\beta}}_{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}!
+\hat{\bm{\beta}}^{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}!
\]
!et
-!split
-===== Friday September 10 =====
+Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics.
+!split
+===== More basic Statistics and Bayes' theorem =====
+
+A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.
+Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.
+
+Assume we have two domains of events $X=[x_0,x_1,\dots,x_{n-1}]$ and $Y=[y_0,y_1,\dots,y_{n-1}]$.
+
+We define also the likelihood for $X$ and $Y$ as $p(X)$ and $p(Y)$ respectively.
+The likelihood of a specific event $x_i$ (or $y_i$) is then written as $p(X=x_i)$ or just $p(x_i)=p_i$.
+
+!bblock Union of events is given by
+!bt
+\[
+p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
+\]
+!et
+!eblock
+
+
+!bblock The product rule is given by
+!bt
+\[
+p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x),
+\]
+!et
+where we read $p(X\vert Y)$ as the likelihood of obtaining $X$ given $Y$.
+!eblock
+
+
+
+!split
+===== Frliday September 10 =====
+
+Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.
!split