From 0650414b411b42c5ea36ec52c6644e8884c218e8 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 9 Sep 2021 07:56:51 +0200 Subject: [PATCH] adding Bayes --- doc/pub/week36/html/week36-bs.html | 65 ++++++++++--------- doc/pub/week36/html/week36-reveal.html | 54 +++++++++++++-- doc/pub/week36/html/week36-solarized.html | 60 +++++++++++++++-- doc/pub/week36/html/week36.html | 60 +++++++++++++++-- doc/pub/week36/ipynb/ipynb-week36-src.tar.gz | Bin 192 -> 191 bytes doc/pub/week36/ipynb/week36.ipynb | 61 +++++++++++++++-- doc/src/week36/week36.do.txt | 46 +++++++++++-- 7 files changed, 288 insertions(+), 58 deletions(-) diff --git a/doc/pub/week36/html/week36-bs.html b/doc/pub/week36/html/week36-bs.html index b4a6abdb4..d7cdc9232 100644 --- a/doc/pub/week36/html/week36-bs.html +++ b/doc/pub/week36/html/week36-bs.html @@ -125,7 +125,11 @@ Automatically generated HTML file from DocOnce source None, 'maximum-likelihood-estimation-mle'), ('A new Cost Function', 2, None, 'a-new-cost-function'), - ('Friday September 10', 2, None, 'friday-september-10'), + ("More basic Statistics and Bayes' theorem", + 2, + None, + 'more-basic-statistics-and-bayes-theorem'), + ('Frliday September 10', 2, None, 'frliday-september-10'), ('Why resampling methods', 2, None, 'why-resampling-methods'), ('Resampling methods', 2, None, 'resampling-methods'), ('Resampling approaches can be computationally expensive', @@ -282,34 +286,35 @@ MathJax.Hub.Config({
  • Independent and Identically Distrubuted (iid)
  • Maximum Likelihood Estimation (MLE)
  • A new Cost Function
    • -
  • Friday September 10
    • -
  • Why resampling methods
    • -
  • Resampling methods
    • -
  • Resampling approaches can be computationally expensive
    • -
  • Why resampling methods ?
    • -
  • Statistical analysis
    • -
  • Resampling methods
    • -
  • Resampling methods: Jackknife and Bootstrap
    • -
  • Resampling methods: Jackknife
    • -
  • Jackknife code example
    • -
  • Resampling methods: Bootstrap
    • -
  • Resampling methods: Bootstrap background
    • -
  • Resampling methods: More Bootstrap background
    • -
  • Resampling methods: Bootstrap approach
    • -
  • Resampling methods: Bootstrap steps
    • -
  • Code example for the Bootstrap method
    • -
  • Various steps in cross-validation
    • -
  • How to set up the cross-validation for Ridge and/or Lasso
    • -
  • Cross-validation in brief
    • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • -
  • The bias-variance tradeoff
    • -
  • Example code for Bias-Variance tradeoff
    • -
  • Understanding what happens
    • -
  • Summing up
    • -
  • Another Example from Scikit-Learn's Repository
    • -
  • More examples on bootstrap and cross-validation and errors
    • -
  • The same example but now with cross-validation
    • -
  • Cross-validation with Ridge
    • +
  • More basic Statistics and Bayes' theorem
    • +
  • Frliday September 10
    • +
  • Why resampling methods
    • +
  • Resampling methods
    • +
  • Resampling approaches can be computationally expensive
    • +
  • Why resampling methods ?
    • +
  • Statistical analysis
    • +
  • Resampling methods
    • +
  • Resampling methods: Jackknife and Bootstrap
    • +
  • Resampling methods: Jackknife
    • +
  • Jackknife code example
    • +
  • Resampling methods: Bootstrap
    • +
  • Resampling methods: Bootstrap background
    • +
  • Resampling methods: More Bootstrap background
    • +
  • Resampling methods: Bootstrap approach
    • +
  • Resampling methods: Bootstrap steps
    • +
  • Code example for the Bootstrap method
    • +
  • Various steps in cross-validation
    • +
  • How to set up the cross-validation for Ridge and/or Lasso
    • +
  • Cross-validation in brief
    • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • +
  • The bias-variance tradeoff
    • +
  • Example code for Bias-Variance tradeoff
    • +
  • Understanding what happens
    • +
  • Summing up
    • +
  • Another Example from Scikit-Learn's Repository
    • +
  • More examples on bootstrap and cross-validation and errors
    • +
  • The same example but now with cross-validation
    • +
  • Cross-validation with Ridge
    • @@ -368,7 +373,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 60
    • +
  • 61
  • »
    • diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html index 510e721e1..2e790d0d5 100644 --- a/doc/pub/week36/html/week36-reveal.html +++ b/doc/pub/week36/html/week36-reveal.html @@ -731,8 +731,8 @@ $$

     

    -There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). -Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). +There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).

    To see this, let us write the cost function for Ridge regression. @@ -1173,17 +1173,61 @@ $$ $$

     
    -which leads to +which leads to the well-known OLS equation for the optimal paramters \( \beta \)

     
    $$ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! $$

     
    + +

    +Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics.

    -

    Friday September 10

    +

    More basic Statistics and Bayes' theorem

    + +

    +A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. + +

    +Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). + +

    +We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). + +

    +

    +Union of events is given by +

     
    +$$ +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +$$ +

     
    +

    + +

    +

    +The product rule is given by +

     
    +$$ +p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x), +$$ +

     
    + +where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). +

    +
    + + +
    +

    Frliday September 10

    + +

    +Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.

    diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html index cbb0689ab..278882d0d 100644 --- a/doc/pub/week36/html/week36-solarized.html +++ b/doc/pub/week36/html/week36-solarized.html @@ -145,7 +145,11 @@ div { text-align: justify; text-justify: inter-word; } None, 'maximum-likelihood-estimation-mle'), ('A new Cost Function', 2, None, 'a-new-cost-function'), - ('Friday September 10', 2, None, 'friday-september-10'), + ("More basic Statistics and Bayes' theorem", + 2, + None, + 'more-basic-statistics-and-bayes-theorem'), + ('Frliday September 10', 2, None, 'frliday-september-10'), ('Why resampling methods', 2, None, 'why-resampling-methods'), ('Resampling methods', 2, None, 'resampling-methods'), ('Resampling approaches can be computationally expensive', @@ -761,8 +765,8 @@ $$ $$

    -There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). -Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). +There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).

    To see this, let us write the cost function for Ridge regression. @@ -1148,15 +1152,59 @@ $$ \boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, $$ -which leads to +which leads to the well-known OLS equation for the optimal paramters \( \beta \) $$ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! $$ +

    +Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. +











    -

    Friday September 10

    +

    More basic Statistics and Bayes' theorem

    + +

    +A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. + +

    +Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). + +

    +We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). + +

    +

    +Union of events is given by +

    +$$ +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +$$ +

    + + +

    +

    +The product rule is given by +

    +$$ +p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x), +$$ + +where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). +

    + + +

    +









    + +

    Frliday September 10

    + +

    +Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.











    diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index 0a067dd6e..d44b27577 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -150,7 +150,11 @@ div { text-align: justify; text-justify: inter-word; } None, 'maximum-likelihood-estimation-mle'), ('A new Cost Function', 2, None, 'a-new-cost-function'), - ('Friday September 10', 2, None, 'friday-september-10'), + ("More basic Statistics and Bayes' theorem", + 2, + None, + 'more-basic-statistics-and-bayes-theorem'), + ('Frliday September 10', 2, None, 'frliday-september-10'), ('Why resampling methods', 2, None, 'why-resampling-methods'), ('Resampling methods', 2, None, 'resampling-methods'), ('Resampling approaches can be computationally expensive', @@ -766,8 +770,8 @@ $$ $$

    -There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). -Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). +There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \).

    To see this, let us write the cost function for Ridge regression. @@ -1153,15 +1157,59 @@ $$ \boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, $$ -which leads to +which leads to the well-known OLS equation for the optimal paramters \( \beta \) $$ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! $$ +

    +Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. +











    -

    Friday September 10

    +

    More basic Statistics and Bayes' theorem

    + +

    +A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. + +

    +Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). + +

    +We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). + +

    +

    +Union of events is given by +

    +$$ +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +$$ +

    + + +

    +

    +The product rule is given by +

    +$$ +p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x), +$$ + +where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). +

    + + +

    +









    + +

    Frliday September 10

    + +

    +Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.











    diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index 45b8b91ed420da6847940ed2d88a7cef2046b621..89a5af289fa95ad824c554339356ce2b8061d263 100644 GIT binary patch literal 191 zcmV;w06_mAiwFP~qB&sz1MSbv3c@f92k@Qu6nTQt?4vsidhj5K_yQfvT%ESscIe*T zeSn@6FGGa>E`LHoNOr5maOOVA!dLnB2QvuF$ac8hIs&N zzLQ>B>xAh~IoC&0mejkseyplK?3rGHXa0$OEiJ6KovXD1r5zPoH*h0r5+2D*rwS-F t*`Wo7uB|i#LUka@QaY)XIE5FZ4@R7A6#n`d&+|O*YY$F{qg((8001d!RsR40 literal 192 zcmV;x06+g9iwFQJn>k?s1MSbv3c@f92k@Qu6nTQtZtXk^?%+WX@dY}TxjNU*wnO*! z?gR9sco`z}cli?%LUPE~n_U*Uy9*XW2uT@(F;^*{lEia8p_Btl$u|b&^K1?Ntti uMmx5^;I)$mL8u-?QAj7X5|^+w`ea07qwv?yc%J8ZUwZ&9oL(6K2mk={H(Emg diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 3f50ff006..bdb0e54f0 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -884,8 +884,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2 via the parameter $\\lambda$.\n", - "Let us for simplicity assume now that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\\beta$ and $\\lambda$.\n", + "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", + "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n", "\n", "To see this, let us write the cost function for Ridge regression. \n", "\n", @@ -1503,7 +1503,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "which leads to" + "which leads to the well-known OLS equation for the optimal paramters $\\beta$" ] }, { @@ -1511,7 +1511,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", + "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", "$$" ] }, @@ -1519,8 +1519,59 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Friday September 10\n", + "Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. \n", "\n", + "## More basic Statistics and Bayes' theorem\n", + "\n", + "A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.\n", + "Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.\n", + "\n", + "Assume we have two domains of events $X=[x_0,x_1,\\dots,x_{n-1}]$ and $Y=[y_0,y_1,\\dots,y_{n-1}]$.\n", + "\n", + "We define also the likelihood for $X$ and $Y$ as $p(X)$ and $p(Y)$ respectively.\n", + "The likelihood of a specific event $x_i$ (or $y_i$) is then written as $p(X=x_i)$ or just $p(x_i)=p_i$. \n", + "\n", + "**Union of events is given by.**" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**The product rule is given by.**" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p(X \\cup Y)= p(X,y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(x),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Frliday September 10\n", + "\n", + "Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1.\n", "\n", "\n", "## Why resampling methods\n", diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index 047fe5215..d5a3d8dc7 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -485,8 +485,8 @@ Inserting the above values we obtain that \] !et -There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2 via the parameter $\lambda$. -Let us for simplicity assume now that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\beta$ and $\lambda$. +There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2$ via the parameter $\lambda$. +Let us for simplicity assume that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\beta$ and $\lambda$. To see this, let us write the cost function for Ridge regression. @@ -853,17 +853,51 @@ Taking the derivative of the *new* cost function with respect to the parameters \bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right) =0, \] !et -which leads to +which leads to the well-known OLS equation for the optimal paramters $\beta$ !bt \[ -\hat{\bm{\beta}}_{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}! +\hat{\bm{\beta}}^{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}! \] !et -!split -===== Friday September 10 ===== +Before we make a similar analysis for Ridge and Lasso regression, we need a new reminder on statistics. +!split +===== More basic Statistics and Bayes' theorem ===== + +A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. + +Assume we have two domains of events $X=[x_0,x_1,\dots,x_{n-1}]$ and $Y=[y_0,y_1,\dots,y_{n-1}]$. + +We define also the likelihood for $X$ and $Y$ as $p(X)$ and $p(Y)$ respectively. +The likelihood of a specific event $x_i$ (or $y_i$) is then written as $p(X=x_i)$ or just $p(x_i)=p_i$. + +!bblock Union of events is given by +!bt +\[ +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +\] +!et +!eblock + + +!bblock The product rule is given by +!bt +\[ +p(X \cup Y)= p(X,y)= p(X\vert Y)p(Y)=p(Y\vert X)p(x), +\] +!et +where we read $p(X\vert Y)$ as the likelihood of obtaining $X$ given $Y$. +!eblock + + + +!split +===== Frliday September 10 ===== + +Depending on where we end on Thursday September 9, we will revise the material for Friday. On Friday we will also discuss project 1. !split