beta--> theta
This commit is contained in:
Morten Hjorth-Jensen
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
@@ -391,7 +391,7 @@ document.write(`
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#assumptions-made">Assumptions made</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance">Expectation value and variance</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-theta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#deriving-ols-from-a-probability-distribution">Deriving OLS from a probability distribution</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#independent-and-identically-distributed-iid">Independent and Identically Distributed (iid)</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -474,7 +474,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
<span class="math notranslate nohighlight">\(\beta\)</span>.</p>
|
||||
<span class="math notranslate nohighlight">\(\theta\)</span>.</p>
|
||||
<p>On of the advantages of doing linear regression is that we actually end up with
|
||||
analytical expressions for several statistical quantities.<br />
|
||||
Standard least squares and Ridge regression allow us to
|
||||
@@ -494,7 +494,7 @@ independent, i.e.:</p>
|
||||
<p>The randomness of <span class="math notranslate nohighlight">\(\varepsilon_i\)</span> implies that
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is also a random variable. In particular,
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is normally distributed, because <span class="math notranslate nohighlight">\(\varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\)</span> is a
|
||||
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\theta}\)</span> is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> we need to calculate its first two moments.</p>
|
||||
<p>Recall that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is a matrix of dimensionality <span class="math notranslate nohighlight">\(n\times p\)</span>. The
|
||||
@@ -514,7 +514,7 @@ which describe our data</p>
|
||||
function <span class="math notranslate nohighlight">\(f\)</span> is approximated by <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> where we want to minimize <span class="math notranslate nohighlight">\((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\)</span>, our MSE, with</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
|
||||
\]</div>
|
||||
</section>
|
||||
<section id="expectation-value-and-variance">
|
||||
@@ -524,8 +524,8 @@ function <span class="math notranslate nohighlight">\(f\)</span> is approximated
|
||||
\[
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
\]</div>
|
||||
<p>while
|
||||
@@ -535,83 +535,83 @@ its variance is</p>
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
\end{split}\]</div>
|
||||
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
|
||||
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\beta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
|
||||
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
|
||||
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\theta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
|
||||
</section>
|
||||
<section id="expectation-value-and-variance-for-boldsymbol-beta">
|
||||
<h2>Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span><a class="headerlink" href="#expectation-value-and-variance-for-boldsymbol-beta" title="Link to this heading">#</a></h2>
|
||||
<p>With the OLS expressions for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}}\)</span> we can evaluate the expectation value</p>
|
||||
<section id="expectation-value-and-variance-for-boldsymbol-theta">
|
||||
<h2>Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#expectation-value-and-variance-for-boldsymbol-theta" title="Link to this heading">#</a></h2>
|
||||
<p>With the OLS expressions for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\theta}}\)</span> we can evaluate the expectation value</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
|
||||
\mathbb{E}(\boldsymbol{\hat{\theta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
|
||||
\]</div>
|
||||
<p>This means that the estimator of the regression parameters is unbiased.</p>
|
||||
<p>We can also calculate the variance</p>
|
||||
<p>The variance of the optimal value <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}}\)</span> is</p>
|
||||
<p>The variance of the optimal value <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\theta}}\)</span> is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
|
||||
\mbox{Var}(\boldsymbol{\hat{\theta}}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
|
||||
\\
|
||||
& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
\end{split}\]</div>
|
||||
<p>where we have used that <span class="math notranslate nohighlight">\(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1}\)</span>, one obtains an estimate of the
|
||||
variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient:
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
|
||||
construct a confidence interval for the estimates.</p>
|
||||
<p>In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and their variance
|
||||
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.</p>
|
||||
<p>It is rather straightforward to show that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
|
||||
\]</div>
|
||||
<p>We see clearly that
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda > 0\)</span>. We say then that the ridge estimator is biased.</p>
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda > 0\)</span>. We say then that the ridge estimator is biased.</p>
|
||||
<p>We can also compute the variance as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\]</div>
|
||||
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> goes to zero.</p>
|
||||
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> goes to zero.</p>
|
||||
<p>With this, we can compute the difference</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\]</div>
|
||||
<p>The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda > 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
|
||||
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda > 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
|
||||
</section>
|
||||
<section id="deriving-ols-from-a-probability-distribution">
|
||||
<h2>Deriving OLS from a probability distribution<a class="headerlink" href="#deriving-ols-from-a-probability-distribution" title="Link to this heading">#</a></h2>
|
||||
@@ -621,14 +621,14 @@ that our output is determined by a given continuous function
|
||||
distribution with zero mean value and an undetermined variance
|
||||
<span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
|
||||
<p>We found above that the outputs <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> have a mean value given by
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span>. This means that a
|
||||
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span>. This means that a
|
||||
single output <span class="math notranslate nohighlight">\(y_i\)</span> is given by the Gaussian distribution</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]</div>
|
||||
</section>
|
||||
<section id="independent-and-identically-distributed-iid">
|
||||
@@ -637,13 +637,13 @@ y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\
|
||||
We define this distribution as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
|
||||
\]</div>
|
||||
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
|
||||
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
|
||||
\]</div>
|
||||
<p>We will write this in a more compact form reserving <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> for the domain of events, including the ouputs (targets) and the inputs. That is
|
||||
in case we have a simple one-dimensional input and output case</p>
|
||||
@@ -655,9 +655,9 @@ in case we have a simple one-dimensional input and output case</p>
|
||||
We can now rewrite the above probability as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]</div>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
|
||||
</section>
|
||||
<section id="maximum-likelihood-estimation-mle">
|
||||
<h2>Maximum Likelihood Estimation (MLE)<a class="headerlink" href="#maximum-likelihood-estimation-mle" title="Link to this heading">#</a></h2>
|
||||
@@ -667,7 +667,7 @@ given some observed data. This is achieved by maximizing a likelihood
|
||||
function so that, under the assumed statistical model, the observed
|
||||
data is the most probable.</p>
|
||||
<p>We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> by
|
||||
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\theta\)</span> by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.</p>
|
||||
@@ -684,22 +684,22 @@ is equivalent to the maximization/minimization of the function itself.</p>
|
||||
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
|
||||
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
|
||||
\]</div>
|
||||
<p>which becomes</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
\]</div>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\beta\)</span> we recognize our familiar OLS equation, namely</p>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\theta\)</span> we recognize our familiar OLS equation, namely</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
|
||||
\]</div>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\beta\)</span></p>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\theta\)</span></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\]</div>
|
||||
<p>Next week we will make a similar analysis for Ridge and Lasso regression</p>
|
||||
</section>
|
||||
@@ -926,23 +926,23 @@ finite <span class="math notranslate nohighlight">\(m\)</span>, it is not always
|
||||
<h2>Confidence Intervals<a class="headerlink" href="#confidence-intervals" title="Link to this heading">#</a></h2>
|
||||
<p>Confidence intervals are used in statistics and represent a type of estimate
|
||||
computed from the observed data. This gives a range of values for an
|
||||
unknown parameter such as the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> from linear regression.</p>
|
||||
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we found
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta}\)</span>, which means that the estimator of the regression parameters is unbiased.</p>
|
||||
unknown parameter such as the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> from linear regression.</p>
|
||||
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we found
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E}(\boldsymbol{\theta}) = \boldsymbol{\theta}\)</span>, which means that the estimator of the regression parameters is unbiased.</p>
|
||||
<p>In the exercises this week we show that the variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient is
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>.</p>
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>.</p>
|
||||
<p>This quantity can be used to
|
||||
construct a confidence interval for the estimates.</p>
|
||||
</section>
|
||||
<section id="standard-approach-based-on-the-normal-distribution">
|
||||
<h2>Standard Approach based on the Normal Distribution<a class="headerlink" href="#standard-approach-based-on-the-normal-distribution" title="Link to this heading">#</a></h2>
|
||||
<p>We will assume that the parameters <span class="math notranslate nohighlight">\(\beta\)</span> follow a normal
|
||||
<p>We will assume that the parameters <span class="math notranslate nohighlight">\(\theta\)</span> follow a normal
|
||||
distribution. We can then define the confidence interval. Here we will be using as
|
||||
shorthands <span class="math notranslate nohighlight">\(\mu_{\beta}\)</span> for the above mean value and <span class="math notranslate nohighlight">\(\sigma_{\beta}\)</span>
|
||||
shorthands <span class="math notranslate nohighlight">\(\mu_{\theta}\)</span> for the above mean value and <span class="math notranslate nohighlight">\(\sigma_{\theta}\)</span>
|
||||
for the standard deviation. We have then a confidence interval</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\left(\mu_{\beta}\pm \frac{z\sigma_{\beta}}{\sqrt{n}}\right),
|
||||
\left(\mu_{\theta}\pm \frac{z\sigma_{\theta}}{\sqrt{n}}\right),
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(z\)</span> defines the level of certainty (or confidence). For a normal
|
||||
distribution typical parameters are <span class="math notranslate nohighlight">\(z=2.576\)</span> which corresponds to a
|
||||
@@ -956,11 +956,11 @@ Bootstrap method, why it works and various theorems related to it.</p>
|
||||
</section>
|
||||
<section id="resampling-methods-bootstrap-background">
|
||||
<h2>Resampling methods: Bootstrap background<a class="headerlink" href="#resampling-methods-bootstrap-background" title="Link to this heading">#</a></h2>
|
||||
<p>Since <span class="math notranslate nohighlight">\(\widehat{\beta} = \widehat{\beta}(\boldsymbol{X})\)</span> is a function of random variables,
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> itself must be a random variable. Thus it has
|
||||
<p>Since <span class="math notranslate nohighlight">\(\widehat{\theta} = \widehat{\theta}(\boldsymbol{X})\)</span> is a function of random variables,
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> itself must be a random variable. Thus it has
|
||||
a pdf, call this function <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. The aim of the bootstrap is to
|
||||
estimate <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> by the relative frequency of
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>. You can think of this as using a histogram
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>. You can think of this as using a histogram
|
||||
in the place of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. If the relative frequency closely
|
||||
resembles <span class="math notranslate nohighlight">\(p(\vec{t})\)</span>, then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> using point
|
||||
@@ -968,18 +968,18 @@ estimators.</p>
|
||||
</section>
|
||||
<section id="resampling-methods-more-bootstrap-background">
|
||||
<h2>Resampling methods: More Bootstrap background<a class="headerlink" href="#resampling-methods-more-bootstrap-background" title="Link to this heading">#</a></h2>
|
||||
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> has
|
||||
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of <span class="math notranslate nohighlight">\(X_i\)</span>, <span class="math notranslate nohighlight">\(p(x)\)</span>, had been known, then it would have
|
||||
been straightforward to do this by:</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Drawing lots of numbers from <span class="math notranslate nohighlight">\(p(x)\)</span>, suppose we call one such set of numbers <span class="math notranslate nohighlight">\((X_1^*, X_2^*, \cdots, X_n^*)\)</span>.</p></li>
|
||||
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>.</p></li>
|
||||
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>.</p></li>
|
||||
</ol>
|
||||
<p>By repeated use of the above two points, many
|
||||
estimates of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> can be obtained. The
|
||||
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>
|
||||
estimates of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> can be obtained. The
|
||||
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>
|
||||
(think of a histogram) as an estimate of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>.</p>
|
||||
</section>
|
||||
<section id="resampling-methods-bootstrap-approach">
|
||||
@@ -1000,18 +1000,18 @@ result in some asymptotic sense? The answer is yes.</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Draw with replacement <span class="math notranslate nohighlight">\(n\)</span> numbers for the observed variables <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,x_n)\)</span>.</p></li>
|
||||
<li><p>Define a vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> containing the values which were drawn from <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>.</p></li>
|
||||
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \beta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
|
||||
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \theta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
|
||||
<li><p>Repeat this process <span class="math notranslate nohighlight">\(k\)</span> times.</p></li>
|
||||
</ol>
|
||||
<p>When you are done, you can draw a histogram of the relative frequency
|
||||
of <span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>. This is your estimate of the probability
|
||||
of <span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>. This is your estimate of the probability
|
||||
distribution <span class="math notranslate nohighlight">\(p(t)\)</span>. Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>. Instead
|
||||
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>. Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of <span class="math notranslate nohighlight">\(\widehat
|
||||
\beta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
|
||||
<span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>.</p>
|
||||
\theta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
|
||||
<span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>.</p>
|
||||
</section>
|
||||
<section id="code-example-for-the-bootstrap-method">
|
||||
<h2>Code example for the Bootstrap method<a class="headerlink" href="#code-example-for-the-bootstrap-method" title="Link to this heading">#</a></h2>
|
||||
@@ -1096,12 +1096,12 @@ tasks. Consider a dataset <span class="math notranslate nohighlight">\(\mathcal{
|
||||
<p>where <span class="math notranslate nohighlight">\(\epsilon\)</span> is normally distributed with mean zero and standard deviation <span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
|
||||
<p>In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function <span class="math notranslate nohighlight">\(f\)</span> in terms of the parameters
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
|
||||
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>Thereafter we found the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> by optimizing the means squared error via the so-called cost function</p>
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
|
||||
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta}\)</span>.</p>
|
||||
<p>Thereafter we found the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> by optimizing the means squared error via the so-called cost function</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
\]</div>
|
||||
<p>We can rewrite this as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -1734,7 +1734,7 @@ the jupyter-notebook from week 37 (September 12-16).</p>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#assumptions-made">Assumptions made</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance">Expectation value and variance</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-theta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#deriving-ols-from-a-probability-distribution">Deriving OLS from a probability distribution</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#independent-and-identically-distributed-iid">Independent and Identically Distributed (iid)</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+197
-197
File diff suppressed because it is too large
Load Diff
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -259,7 +259,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
\( \beta \).
|
||||
\( \theta \).
|
||||
</p>
|
||||
|
||||
<p>On of the advantages of doing linear regression is that we actually end up with
|
||||
@@ -284,7 +284,7 @@ $$
|
||||
<p>The randomness of \( \varepsilon_i \) implies that
|
||||
\( \mathbf{y}_i \) is also a random variable. In particular,
|
||||
\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\theta} \) is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
\( \mathbf{y}_i \) we need to calculate its first two moments.
|
||||
</p>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -265,7 +265,7 @@ $$
|
||||
function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with
|
||||
</p>
|
||||
$$
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
|
||||
$$
|
||||
|
||||
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -257,8 +257,8 @@ MathJax.Hub.Config({
|
||||
$$
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
@@ -269,19 +269,19 @@ $$
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\theta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
</p>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -251,81 +251,81 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
<a name="part0006"></a>
|
||||
<!-- !split -->
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-beta" class="anchor">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-theta" class="anchor">Expectation value and variance for \( \boldsymbol{\theta} \) </h2>
|
||||
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\beta}} \) we can evaluate the expectation value</p>
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\theta}} \) we can evaluate the expectation value</p>
|
||||
$$
|
||||
\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
|
||||
\mathbb{E}(\boldsymbol{\hat{\theta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
|
||||
$$
|
||||
|
||||
<p>This means that the estimator of the regression parameters is unbiased.</p>
|
||||
|
||||
<p>We can also calculate the variance</p>
|
||||
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\beta}} \) is</p>
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\theta}} \) is</p>
|
||||
$$
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
|
||||
\mbox{Var}(\boldsymbol{\hat{\theta}}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
|
||||
\\
|
||||
& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
$$
|
||||
|
||||
<p>where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the
|
||||
variance of the estimate of the \( j \)-th regression coefficient:
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
construct a confidence interval for the estimates.
|
||||
</p>
|
||||
|
||||
<p>In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters \( \boldsymbol{\beta} \) and their variance
|
||||
expectation values of the parameters \( \boldsymbol{\theta} \) and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.
|
||||
</p>
|
||||
|
||||
<p>It is rather straightforward to show that</p>
|
||||
$$
|
||||
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
|
||||
$$
|
||||
|
||||
<p>We see clearly that
|
||||
\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
\( \mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
</p>
|
||||
|
||||
<p>We can also compute the variance as </p>
|
||||
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
$$
|
||||
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. </p>
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\theta} \) goes to zero. </p>
|
||||
|
||||
<p>With this, we can compute the difference </p>
|
||||
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
$$
|
||||
|
||||
<p>The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\theta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -261,15 +261,15 @@ distribution with zero mean value and an undetermined variance
|
||||
</p>
|
||||
|
||||
<p>We found above that the outputs \( \boldsymbol{y} \) have a mean value given by
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\theta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\theta}} \). This means that a
|
||||
single output \( y_i \) is given by the Gaussian distribution
|
||||
</p>
|
||||
|
||||
$$
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
|
||||
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -257,15 +257,15 @@ MathJax.Hub.Config({
|
||||
We define this distribution as
|
||||
</p>
|
||||
$$
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
|
||||
$$
|
||||
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).</p>
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\theta} \).</p>
|
||||
|
||||
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have</p>
|
||||
|
||||
$$
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
|
||||
$$
|
||||
|
||||
<p>We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is
|
||||
@@ -279,10 +279,10 @@ $$
|
||||
We can now rewrite the above probability as
|
||||
</p>
|
||||
$$
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).</p>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\theta} \).</p>
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -261,7 +261,7 @@ data is the most probable.
|
||||
</p>
|
||||
|
||||
<p>We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters \( \beta \) by
|
||||
distribution and we will determine the optimal parameters \( \theta \) by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -256,23 +256,23 @@ MathJax.Hub.Config({
|
||||
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
|
||||
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
|
||||
$$
|
||||
|
||||
<p>which becomes</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
$$
|
||||
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely</p>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \theta \) we recognize our familiar OLS equation, namely</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
|
||||
$$
|
||||
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \beta \)</p>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \theta \)</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
$$
|
||||
|
||||
<p>Next week we will make a similar analysis for Ridge and Lasso regression</p>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -255,15 +255,15 @@ MathJax.Hub.Config({
|
||||
|
||||
<p>Confidence intervals are used in statistics and represent a type of estimate
|
||||
computed from the observed data. This gives a range of values for an
|
||||
unknown parameter such as the parameters \( \boldsymbol{\beta} \) from linear regression.
|
||||
unknown parameter such as the parameters \( \boldsymbol{\theta} \) from linear regression.
|
||||
</p>
|
||||
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\theta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\theta}) = \boldsymbol{\theta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
</p>
|
||||
|
||||
<p>In the exercises this week we show that the variance of the estimate of the \( j \)-th regression coefficient is
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
</p>
|
||||
|
||||
<p>This quantity can be used to
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -253,14 +253,14 @@ MathJax.Hub.Config({
|
||||
<!-- !split -->
|
||||
<h2 id="standard-approach-based-on-the-normal-distribution" class="anchor">Standard Approach based on the Normal Distribution </h2>
|
||||
|
||||
<p>We will assume that the parameters \( \beta \) follow a normal
|
||||
<p>We will assume that the parameters \( \theta \) follow a normal
|
||||
distribution. We can then define the confidence interval. Here we will be using as
|
||||
shorthands \( \mu_{\beta} \) for the above mean value and \( \sigma_{\beta} \)
|
||||
shorthands \( \mu_{\theta} \) for the above mean value and \( \sigma_{\theta} \)
|
||||
for the standard deviation. We have then a confidence interval
|
||||
</p>
|
||||
|
||||
$$
|
||||
\left(\mu_{\beta}\pm \frac{z\sigma_{\beta}}{\sqrt{n}}\right),
|
||||
\left(\mu_{\theta}\pm \frac{z\sigma_{\theta}}{\sqrt{n}}\right),
|
||||
$$
|
||||
|
||||
<p>where \( z \) defines the level of certainty (or confidence). For a normal
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -253,11 +253,11 @@ MathJax.Hub.Config({
|
||||
<!-- !split -->
|
||||
<h2 id="resampling-methods-bootstrap-background" class="anchor">Resampling methods: Bootstrap background </h2>
|
||||
|
||||
<p>Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\beta} \) itself must be a random variable. Thus it has
|
||||
<p>Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\theta} \) itself must be a random variable. Thus it has
|
||||
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
|
||||
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
|
||||
\( \widehat{\beta} \). You can think of this as using a histogram
|
||||
\( \widehat{\theta} \). You can think of this as using a histogram
|
||||
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
|
||||
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -253,7 +253,7 @@ MathJax.Hub.Config({
|
||||
<!-- !split -->
|
||||
<h2 id="resampling-methods-more-bootstrap-background" class="anchor">Resampling methods: More Bootstrap background </h2>
|
||||
|
||||
<p>In the case that \( \widehat{\beta} \) has
|
||||
<p>In the case that \( \widehat{\theta} \) has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of \( X_i \), \( p(x) \), had been known, then it would have
|
||||
@@ -261,11 +261,11 @@ been straightforward to do this by:
|
||||
</p>
|
||||
<ol>
|
||||
<li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
|
||||
<li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
|
||||
<li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
|
||||
</ol>
|
||||
<p>By repeated use of the above two points, many
|
||||
estimates of \( \widehat{\beta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\beta}^* \)
|
||||
estimates of \( \widehat{\theta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\theta}^* \)
|
||||
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
|
||||
</p>
|
||||
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -258,18 +258,18 @@ MathJax.Hub.Config({
|
||||
<ol>
|
||||
<li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
|
||||
<li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
|
||||
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<li> Repeat this process \( k \) times.</li>
|
||||
</ol>
|
||||
<p>When you are done, you can draw a histogram of the relative frequency
|
||||
of \( \widehat \beta^* \). This is your estimate of the probability
|
||||
of \( \widehat \theta^* \). This is your estimate of the probability
|
||||
distribution \( p(t) \). Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
|
||||
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of \( \widehat
|
||||
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \beta^* \).
|
||||
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \theta^* \).
|
||||
</p>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -270,13 +270,13 @@ $$
|
||||
|
||||
<p>In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function \( f \) in terms of the parameters
|
||||
\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \).
|
||||
\( \boldsymbol{\theta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\theta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
$$
|
||||
|
||||
<p>We can rewrite this as </p>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -51,10 +51,10 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs006.html#expectation-value-and-variance-for-boldsymbol-theta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\theta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs007.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs008.html#independent-and-identically-distributed-iid" style="font-size: 80%;">Independent and Identically Distributed (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week38-bs009.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
@@ -233,7 +233,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
\( \beta \).
|
||||
\( \theta \).
|
||||
</p>
|
||||
|
||||
<p>On of the advantages of doing linear regression is that we actually end up with
|
||||
@@ -260,7 +260,7 @@ $$
|
||||
<p>The randomness of \( \varepsilon_i \) implies that
|
||||
\( \mathbf{y}_i \) is also a random variable. In particular,
|
||||
\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\theta} \) is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
\( \mathbf{y}_i \) we need to calculate its first two moments.
|
||||
</p>
|
||||
@@ -289,7 +289,7 @@ function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want t
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
|
||||
$$
|
||||
<p> <br>
|
||||
</section>
|
||||
@@ -302,8 +302,8 @@ $$
|
||||
$$
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
@@ -316,30 +316,30 @@ $$
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\theta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-theta">Expectation value and variance for \( \boldsymbol{\theta} \) </h2>
|
||||
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\beta}} \) we can evaluate the expectation value</p>
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\theta}} \) we can evaluate the expectation value</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
|
||||
\mathbb{E}(\boldsymbol{\hat{\theta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -347,78 +347,78 @@ $$
|
||||
|
||||
<p>We can also calculate the variance</p>
|
||||
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\beta}} \) is</p>
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\theta}} \) is</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
|
||||
\mbox{Var}(\boldsymbol{\hat{\theta}}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
|
||||
\\
|
||||
& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the
|
||||
variance of the estimate of the \( j \)-th regression coefficient:
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
construct a confidence interval for the estimates.
|
||||
</p>
|
||||
|
||||
<p>In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters \( \boldsymbol{\beta} \) and their variance
|
||||
expectation values of the parameters \( \boldsymbol{\theta} \) and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.
|
||||
</p>
|
||||
|
||||
<p>It is rather straightforward to show that</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>We see clearly that
|
||||
\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
\( \mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
</p>
|
||||
|
||||
<p>We can also compute the variance as </p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. </p>
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\theta} \) goes to zero. </p>
|
||||
|
||||
<p>With this, we can compute the difference </p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\theta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
</p>
|
||||
</section>
|
||||
|
||||
@@ -433,16 +433,16 @@ distribution with zero mean value and an undetermined variance
|
||||
</p>
|
||||
|
||||
<p>We found above that the outputs \( \boldsymbol{y} \) have a mean value given by
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\theta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\theta}} \). This means that a
|
||||
single output \( y_i \) is given by the Gaussian distribution
|
||||
</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
<p> <br>
|
||||
</section>
|
||||
@@ -455,17 +455,17 @@ We define this distribution as
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).</p>
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\theta} \).</p>
|
||||
|
||||
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -483,11 +483,11 @@ We can now rewrite the above probability as
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).</p>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\theta} \).</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
@@ -501,7 +501,7 @@ data is the most probable.
|
||||
</p>
|
||||
|
||||
<p>We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters \( \beta \) by
|
||||
distribution and we will determine the optimal parameters \( \theta \) by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.
|
||||
@@ -526,29 +526,29 @@ is equivalent to the maximization/minimization of the function itself.
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
|
||||
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>which becomes</p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely</p>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \theta \) we recognize our familiar OLS equation, namely</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \beta \)</p>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \theta \)</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -879,15 +879,15 @@ finite \( m \), it is not always possible to find a closed form /analytic expres
|
||||
|
||||
<p>Confidence intervals are used in statistics and represent a type of estimate
|
||||
computed from the observed data. This gives a range of values for an
|
||||
unknown parameter such as the parameters \( \boldsymbol{\beta} \) from linear regression.
|
||||
unknown parameter such as the parameters \( \boldsymbol{\theta} \) from linear regression.
|
||||
</p>
|
||||
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\theta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\theta}) = \boldsymbol{\theta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
</p>
|
||||
|
||||
<p>In the exercises this week we show that the variance of the estimate of the \( j \)-th regression coefficient is
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
</p>
|
||||
|
||||
<p>This quantity can be used to
|
||||
@@ -898,15 +898,15 @@ construct a confidence interval for the estimates.
|
||||
<section>
|
||||
<h2 id="standard-approach-based-on-the-normal-distribution">Standard Approach based on the Normal Distribution </h2>
|
||||
|
||||
<p>We will assume that the parameters \( \beta \) follow a normal
|
||||
<p>We will assume that the parameters \( \theta \) follow a normal
|
||||
distribution. We can then define the confidence interval. Here we will be using as
|
||||
shorthands \( \mu_{\beta} \) for the above mean value and \( \sigma_{\beta} \)
|
||||
shorthands \( \mu_{\theta} \) for the above mean value and \( \sigma_{\theta} \)
|
||||
for the standard deviation. We have then a confidence interval
|
||||
</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\left(\mu_{\beta}\pm \frac{z\sigma_{\beta}}{\sqrt{n}}\right),
|
||||
\left(\mu_{\theta}\pm \frac{z\sigma_{\theta}}{\sqrt{n}}\right),
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -928,11 +928,11 @@ Bootstrap method, why it works and various theorems related to it.
|
||||
<section>
|
||||
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
|
||||
|
||||
<p>Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\beta} \) itself must be a random variable. Thus it has
|
||||
<p>Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\theta} \) itself must be a random variable. Thus it has
|
||||
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
|
||||
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
|
||||
\( \widehat{\beta} \). You can think of this as using a histogram
|
||||
\( \widehat{\theta} \). You can think of this as using a histogram
|
||||
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
|
||||
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
|
||||
@@ -943,7 +943,7 @@ estimators.
|
||||
<section>
|
||||
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
|
||||
|
||||
<p>In the case that \( \widehat{\beta} \) has
|
||||
<p>In the case that \( \widehat{\theta} \) has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of \( X_i \), \( p(x) \), had been known, then it would have
|
||||
@@ -951,12 +951,12 @@ been straightforward to do this by:
|
||||
</p>
|
||||
<ol>
|
||||
<p><li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
|
||||
<p><li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
|
||||
<p><li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
|
||||
</ol>
|
||||
<p>
|
||||
<p>By repeated use of the above two points, many
|
||||
estimates of \( \widehat{\beta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\beta}^* \)
|
||||
estimates of \( \widehat{\theta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\theta}^* \)
|
||||
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
|
||||
</p>
|
||||
</section>
|
||||
@@ -986,19 +986,19 @@ result in some asymptotic sense? The answer is yes.
|
||||
<ol>
|
||||
<p><li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
|
||||
<p><li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
|
||||
<p><li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<p><li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<p><li> Repeat this process \( k \) times.</li>
|
||||
</ol>
|
||||
<p>
|
||||
<p>When you are done, you can draw a histogram of the relative frequency
|
||||
of \( \widehat \beta^* \). This is your estimate of the probability
|
||||
of \( \widehat \theta^* \). This is your estimate of the probability
|
||||
distribution \( p(t) \). Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
|
||||
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of \( \widehat
|
||||
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \beta^* \).
|
||||
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \theta^* \).
|
||||
</p>
|
||||
</section>
|
||||
|
||||
@@ -1126,14 +1126,14 @@ $$
|
||||
|
||||
<p>In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function \( f \) in terms of the parameters
|
||||
\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \).
|
||||
\( \boldsymbol{\theta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\theta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
|
||||
@@ -78,10 +78,10 @@ div.toc p,a {
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -270,7 +270,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
\( \beta \).
|
||||
\( \theta \).
|
||||
</p>
|
||||
|
||||
<p>On of the advantages of doing linear regression is that we actually end up with
|
||||
@@ -295,7 +295,7 @@ $$
|
||||
<p>The randomness of \( \varepsilon_i \) implies that
|
||||
\( \mathbf{y}_i \) is also a random variable. In particular,
|
||||
\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\theta} \) is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
\( \mathbf{y}_i \) we need to calculate its first two moments.
|
||||
</p>
|
||||
@@ -320,7 +320,7 @@ $$
|
||||
function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with
|
||||
</p>
|
||||
$$
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
|
||||
$$
|
||||
|
||||
|
||||
@@ -331,8 +331,8 @@ $$
|
||||
$$
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
@@ -343,97 +343,97 @@ $$
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\theta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-theta">Expectation value and variance for \( \boldsymbol{\theta} \) </h2>
|
||||
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\beta}} \) we can evaluate the expectation value</p>
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\theta}} \) we can evaluate the expectation value</p>
|
||||
$$
|
||||
\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
|
||||
\mathbb{E}(\boldsymbol{\hat{\theta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
|
||||
$$
|
||||
|
||||
<p>This means that the estimator of the regression parameters is unbiased.</p>
|
||||
|
||||
<p>We can also calculate the variance</p>
|
||||
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\beta}} \) is</p>
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\theta}} \) is</p>
|
||||
$$
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
|
||||
\mbox{Var}(\boldsymbol{\hat{\theta}}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
|
||||
\\
|
||||
& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
$$
|
||||
|
||||
<p>where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the
|
||||
variance of the estimate of the \( j \)-th regression coefficient:
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
construct a confidence interval for the estimates.
|
||||
</p>
|
||||
|
||||
<p>In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters \( \boldsymbol{\beta} \) and their variance
|
||||
expectation values of the parameters \( \boldsymbol{\theta} \) and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.
|
||||
</p>
|
||||
|
||||
<p>It is rather straightforward to show that</p>
|
||||
$$
|
||||
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
|
||||
$$
|
||||
|
||||
<p>We see clearly that
|
||||
\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
\( \mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
</p>
|
||||
|
||||
<p>We can also compute the variance as </p>
|
||||
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
$$
|
||||
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. </p>
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\theta} \) goes to zero. </p>
|
||||
|
||||
<p>With this, we can compute the difference </p>
|
||||
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
$$
|
||||
|
||||
<p>The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\theta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -447,15 +447,15 @@ distribution with zero mean value and an undetermined variance
|
||||
</p>
|
||||
|
||||
<p>We found above that the outputs \( \boldsymbol{y} \) have a mean value given by
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\theta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\theta}} \). This means that a
|
||||
single output \( y_i \) is given by the Gaussian distribution
|
||||
</p>
|
||||
|
||||
$$
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
|
||||
|
||||
@@ -466,15 +466,15 @@ $$
|
||||
We define this distribution as
|
||||
</p>
|
||||
$$
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
|
||||
$$
|
||||
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).</p>
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\theta} \).</p>
|
||||
|
||||
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have</p>
|
||||
|
||||
$$
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
|
||||
$$
|
||||
|
||||
<p>We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is
|
||||
@@ -488,10 +488,10 @@ $$
|
||||
We can now rewrite the above probability as
|
||||
</p>
|
||||
$$
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).</p>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\theta} \).</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE) </h2>
|
||||
@@ -504,7 +504,7 @@ data is the most probable.
|
||||
</p>
|
||||
|
||||
<p>We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters \( \beta \) by
|
||||
distribution and we will determine the optimal parameters \( \theta \) by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.
|
||||
@@ -527,23 +527,23 @@ is equivalent to the maximization/minimization of the function itself.
|
||||
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
|
||||
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
|
||||
$$
|
||||
|
||||
<p>which becomes</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
$$
|
||||
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely</p>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \theta \) we recognize our familiar OLS equation, namely</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
|
||||
$$
|
||||
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \beta \)</p>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \theta \)</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
$$
|
||||
|
||||
<p>Next week we will make a similar analysis for Ridge and Lasso regression</p>
|
||||
@@ -838,15 +838,15 @@ finite \( m \), it is not always possible to find a closed form /analytic expres
|
||||
|
||||
<p>Confidence intervals are used in statistics and represent a type of estimate
|
||||
computed from the observed data. This gives a range of values for an
|
||||
unknown parameter such as the parameters \( \boldsymbol{\beta} \) from linear regression.
|
||||
unknown parameter such as the parameters \( \boldsymbol{\theta} \) from linear regression.
|
||||
</p>
|
||||
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\theta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\theta}) = \boldsymbol{\theta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
</p>
|
||||
|
||||
<p>In the exercises this week we show that the variance of the estimate of the \( j \)-th regression coefficient is
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
</p>
|
||||
|
||||
<p>This quantity can be used to
|
||||
@@ -856,14 +856,14 @@ construct a confidence interval for the estimates.
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="standard-approach-based-on-the-normal-distribution">Standard Approach based on the Normal Distribution </h2>
|
||||
|
||||
<p>We will assume that the parameters \( \beta \) follow a normal
|
||||
<p>We will assume that the parameters \( \theta \) follow a normal
|
||||
distribution. We can then define the confidence interval. Here we will be using as
|
||||
shorthands \( \mu_{\beta} \) for the above mean value and \( \sigma_{\beta} \)
|
||||
shorthands \( \mu_{\theta} \) for the above mean value and \( \sigma_{\theta} \)
|
||||
for the standard deviation. We have then a confidence interval
|
||||
</p>
|
||||
|
||||
$$
|
||||
\left(\mu_{\beta}\pm \frac{z\sigma_{\beta}}{\sqrt{n}}\right),
|
||||
\left(\mu_{\theta}\pm \frac{z\sigma_{\theta}}{\sqrt{n}}\right),
|
||||
$$
|
||||
|
||||
<p>where \( z \) defines the level of certainty (or confidence). For a normal
|
||||
@@ -883,11 +883,11 @@ Bootstrap method, why it works and various theorems related to it.
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
|
||||
|
||||
<p>Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\beta} \) itself must be a random variable. Thus it has
|
||||
<p>Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\theta} \) itself must be a random variable. Thus it has
|
||||
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
|
||||
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
|
||||
\( \widehat{\beta} \). You can think of this as using a histogram
|
||||
\( \widehat{\theta} \). You can think of this as using a histogram
|
||||
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
|
||||
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
|
||||
@@ -897,7 +897,7 @@ estimators.
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
|
||||
|
||||
<p>In the case that \( \widehat{\beta} \) has
|
||||
<p>In the case that \( \widehat{\theta} \) has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of \( X_i \), \( p(x) \), had been known, then it would have
|
||||
@@ -905,11 +905,11 @@ been straightforward to do this by:
|
||||
</p>
|
||||
<ol>
|
||||
<li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
|
||||
<li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
|
||||
<li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
|
||||
</ol>
|
||||
<p>By repeated use of the above two points, many
|
||||
estimates of \( \widehat{\beta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\beta}^* \)
|
||||
estimates of \( \widehat{\theta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\theta}^* \)
|
||||
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
|
||||
</p>
|
||||
|
||||
@@ -937,18 +937,18 @@ result in some asymptotic sense? The answer is yes.
|
||||
<ol>
|
||||
<li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
|
||||
<li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
|
||||
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<li> Repeat this process \( k \) times.</li>
|
||||
</ol>
|
||||
<p>When you are done, you can draw a histogram of the relative frequency
|
||||
of \( \widehat \beta^* \). This is your estimate of the probability
|
||||
of \( \widehat \theta^* \). This is your estimate of the probability
|
||||
distribution \( p(t) \). Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
|
||||
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of \( \widehat
|
||||
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \beta^* \).
|
||||
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \theta^* \).
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1072,13 +1072,13 @@ $$
|
||||
|
||||
<p>In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function \( f \) in terms of the parameters
|
||||
\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \).
|
||||
\( \boldsymbol{\theta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\theta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
$$
|
||||
|
||||
<p>We can rewrite this as </p>
|
||||
|
||||
@@ -155,10 +155,10 @@ div.toc p,a {
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance'),
|
||||
('Expectation value and variance for $\\boldsymbol{\\beta}$',
|
||||
('Expectation value and variance for $\\boldsymbol{\\theta}$',
|
||||
2,
|
||||
None,
|
||||
'expectation-value-and-variance-for-boldsymbol-beta'),
|
||||
'expectation-value-and-variance-for-boldsymbol-theta'),
|
||||
('Deriving OLS from a probability distribution',
|
||||
2,
|
||||
None,
|
||||
@@ -347,7 +347,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
\( \beta \).
|
||||
\( \theta \).
|
||||
</p>
|
||||
|
||||
<p>On of the advantages of doing linear regression is that we actually end up with
|
||||
@@ -372,7 +372,7 @@ $$
|
||||
<p>The randomness of \( \varepsilon_i \) implies that
|
||||
\( \mathbf{y}_i \) is also a random variable. In particular,
|
||||
\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a
|
||||
\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\theta} \) is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
\( \mathbf{y}_i \) we need to calculate its first two moments.
|
||||
</p>
|
||||
@@ -397,7 +397,7 @@ $$
|
||||
function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with
|
||||
</p>
|
||||
$$
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
|
||||
$$
|
||||
|
||||
|
||||
@@ -408,8 +408,8 @@ $$
|
||||
$$
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
@@ -420,97 +420,97 @@ $$
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
<p>Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
|
||||
mean value \( \boldsymbol{X}\boldsymbol{\theta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
|
||||
<h2 id="expectation-value-and-variance-for-boldsymbol-theta">Expectation value and variance for \( \boldsymbol{\theta} \) </h2>
|
||||
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\beta}} \) we can evaluate the expectation value</p>
|
||||
<p>With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\theta}} \) we can evaluate the expectation value</p>
|
||||
$$
|
||||
\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
|
||||
\mathbb{E}(\boldsymbol{\hat{\theta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
|
||||
$$
|
||||
|
||||
<p>This means that the estimator of the regression parameters is unbiased.</p>
|
||||
|
||||
<p>We can also calculate the variance</p>
|
||||
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\beta}} \) is</p>
|
||||
<p>The variance of the optimal value \( \boldsymbol{\hat{\theta}} \) is</p>
|
||||
$$
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
|
||||
\mbox{Var}(\boldsymbol{\hat{\theta}}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
|
||||
\\
|
||||
& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
$$
|
||||
|
||||
<p>where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the
|
||||
variance of the estimate of the \( j \)-th regression coefficient:
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to
|
||||
construct a confidence interval for the estimates.
|
||||
</p>
|
||||
|
||||
<p>In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters \( \boldsymbol{\beta} \) and their variance
|
||||
expectation values of the parameters \( \boldsymbol{\theta} \) and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.
|
||||
</p>
|
||||
|
||||
<p>It is rather straightforward to show that</p>
|
||||
$$
|
||||
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
|
||||
$$
|
||||
|
||||
<p>We see clearly that
|
||||
\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
\( \mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased.
|
||||
</p>
|
||||
|
||||
<p>We can also compute the variance as </p>
|
||||
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
$$
|
||||
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. </p>
|
||||
<p>and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\theta} \) goes to zero. </p>
|
||||
|
||||
<p>With this, we can compute the difference </p>
|
||||
|
||||
$$
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
$$
|
||||
|
||||
<p>The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\theta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -524,15 +524,15 @@ distribution with zero mean value and an undetermined variance
|
||||
</p>
|
||||
|
||||
<p>We found above that the outputs \( \boldsymbol{y} \) have a mean value given by
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
\( \boldsymbol{X}\hat{\boldsymbol{\theta}} \) and variance \( \sigma^2 \). Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a
|
||||
but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\theta}} \). This means that a
|
||||
single output \( y_i \) is given by the Gaussian distribution
|
||||
</p>
|
||||
|
||||
$$
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
|
||||
|
||||
@@ -543,15 +543,15 @@ $$
|
||||
We define this distribution as
|
||||
</p>
|
||||
$$
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
|
||||
$$
|
||||
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).</p>
|
||||
<p>which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\theta} \).</p>
|
||||
|
||||
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have</p>
|
||||
|
||||
$$
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
|
||||
$$
|
||||
|
||||
<p>We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is
|
||||
@@ -565,10 +565,10 @@ $$
|
||||
We can now rewrite the above probability as
|
||||
</p>
|
||||
$$
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
$$
|
||||
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).</p>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\theta} \).</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE) </h2>
|
||||
@@ -581,7 +581,7 @@ data is the most probable.
|
||||
</p>
|
||||
|
||||
<p>We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters \( \beta \) by
|
||||
distribution and we will determine the optimal parameters \( \theta \) by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.
|
||||
@@ -604,23 +604,23 @@ is equivalent to the maximization/minimization of the function itself.
|
||||
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
|
||||
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
|
||||
$$
|
||||
|
||||
<p>which becomes</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
$$
|
||||
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely</p>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters \( \theta \) we recognize our familiar OLS equation, namely</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
|
||||
$$
|
||||
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \beta \)</p>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters \( \theta \)</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
$$
|
||||
|
||||
<p>Next week we will make a similar analysis for Ridge and Lasso regression</p>
|
||||
@@ -915,15 +915,15 @@ finite \( m \), it is not always possible to find a closed form /analytic expres
|
||||
|
||||
<p>Confidence intervals are used in statistics and represent a type of estimate
|
||||
computed from the observed data. This gives a range of values for an
|
||||
unknown parameter such as the parameters \( \boldsymbol{\beta} \) from linear regression.
|
||||
unknown parameter such as the parameters \( \boldsymbol{\theta} \) from linear regression.
|
||||
</p>
|
||||
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
<p>With the OLS expressions for the parameters \( \boldsymbol{\theta} \) we found
|
||||
\( \mathbb{E}(\boldsymbol{\theta}) = \boldsymbol{\theta} \), which means that the estimator of the regression parameters is unbiased.
|
||||
</p>
|
||||
|
||||
<p>In the exercises this week we show that the variance of the estimate of the \( j \)-th regression coefficient is
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
\( \boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \).
|
||||
</p>
|
||||
|
||||
<p>This quantity can be used to
|
||||
@@ -933,14 +933,14 @@ construct a confidence interval for the estimates.
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="standard-approach-based-on-the-normal-distribution">Standard Approach based on the Normal Distribution </h2>
|
||||
|
||||
<p>We will assume that the parameters \( \beta \) follow a normal
|
||||
<p>We will assume that the parameters \( \theta \) follow a normal
|
||||
distribution. We can then define the confidence interval. Here we will be using as
|
||||
shorthands \( \mu_{\beta} \) for the above mean value and \( \sigma_{\beta} \)
|
||||
shorthands \( \mu_{\theta} \) for the above mean value and \( \sigma_{\theta} \)
|
||||
for the standard deviation. We have then a confidence interval
|
||||
</p>
|
||||
|
||||
$$
|
||||
\left(\mu_{\beta}\pm \frac{z\sigma_{\beta}}{\sqrt{n}}\right),
|
||||
\left(\mu_{\theta}\pm \frac{z\sigma_{\theta}}{\sqrt{n}}\right),
|
||||
$$
|
||||
|
||||
<p>where \( z \) defines the level of certainty (or confidence). For a normal
|
||||
@@ -960,11 +960,11 @@ Bootstrap method, why it works and various theorems related to it.
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
|
||||
|
||||
<p>Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\beta} \) itself must be a random variable. Thus it has
|
||||
<p>Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
|
||||
\( \widehat{\theta} \) itself must be a random variable. Thus it has
|
||||
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
|
||||
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
|
||||
\( \widehat{\beta} \). You can think of this as using a histogram
|
||||
\( \widehat{\theta} \). You can think of this as using a histogram
|
||||
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
|
||||
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
|
||||
@@ -974,7 +974,7 @@ estimators.
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
|
||||
|
||||
<p>In the case that \( \widehat{\beta} \) has
|
||||
<p>In the case that \( \widehat{\theta} \) has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of \( X_i \), \( p(x) \), had been known, then it would have
|
||||
@@ -982,11 +982,11 @@ been straightforward to do this by:
|
||||
</p>
|
||||
<ol>
|
||||
<li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
|
||||
<li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
|
||||
<li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
|
||||
</ol>
|
||||
<p>By repeated use of the above two points, many
|
||||
estimates of \( \widehat{\beta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\beta}^* \)
|
||||
estimates of \( \widehat{\theta} \) can be obtained. The
|
||||
idea is to use the relative frequency of \( \widehat{\theta}^* \)
|
||||
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
|
||||
</p>
|
||||
|
||||
@@ -1014,18 +1014,18 @@ result in some asymptotic sense? The answer is yes.
|
||||
<ol>
|
||||
<li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
|
||||
<li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
|
||||
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
|
||||
<li> Repeat this process \( k \) times.</li>
|
||||
</ol>
|
||||
<p>When you are done, you can draw a histogram of the relative frequency
|
||||
of \( \widehat \beta^* \). This is your estimate of the probability
|
||||
of \( \widehat \theta^* \). This is your estimate of the probability
|
||||
distribution \( p(t) \). Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
|
||||
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of \( \widehat
|
||||
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \beta^* \).
|
||||
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
|
||||
\( \widehat \theta^* \).
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1149,13 +1149,13 @@ $$
|
||||
|
||||
<p>In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function \( f \) in terms of the parameters
|
||||
\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \).
|
||||
\( \boldsymbol{\theta} \) and the design matrix \( \boldsymbol{X} \) which embody our model,
|
||||
that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
<p>Thereafter we found the parameters \( \boldsymbol{\theta} \) by optimizing the means squared error via the so-called cost function</p>
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
$$
|
||||
|
||||
<p>We can rewrite this as </p>
|
||||
|
||||
Binary file not shown.
+197
-197
File diff suppressed because it is too large
Load Diff
@@ -37,7 +37,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
$\beta$.
|
||||
$\theta$.
|
||||
|
||||
|
||||
On of the advantages of doing linear regression is that we actually end up with
|
||||
@@ -60,7 +60,7 @@ independent, i.e.:
|
||||
The randomness of $\varepsilon_i$ implies that
|
||||
$\mathbf{y}_i$ is also a random variable. In particular,
|
||||
$\mathbf{y}_i$ is normally distributed, because $\varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \bm{\beta}$ is a
|
||||
\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \bm{\theta}$ is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
$\mathbf{y}_i$ we need to calculate its first two moments.
|
||||
|
||||
@@ -85,7 +85,7 @@ We approximate this function with our model from the solution of the linear regr
|
||||
function $f$ is approximated by $\bm{\tilde{y}}$ where we want to minimize $(\bm{y}-\bm{\tilde{y}})^2$, our MSE, with
|
||||
!bt
|
||||
\[
|
||||
\bm{\tilde{y}} = \bm{X}\bm{\beta}.
|
||||
\bm{\tilde{y}} = \bm{X}\bm{\theta}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -96,8 +96,8 @@ We can calculate the expectation value of $\bm{y}$ for a given element $i$
|
||||
!bt
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
!et
|
||||
while
|
||||
@@ -106,97 +106,97 @@ its variance is
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \bm{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \bm{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
!et
|
||||
Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
|
||||
mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD).
|
||||
Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\theta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
|
||||
mean value $\bm{X}\bm{\theta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD).
|
||||
|
||||
!split
|
||||
===== Expectation value and variance for $\bm{\beta}$ =====
|
||||
===== Expectation value and variance for $\bm{\theta}$ =====
|
||||
|
||||
With the OLS expressions for the optimal parameters $\bm{\hat{\beta}}$ we can evaluate the expectation value
|
||||
With the OLS expressions for the optimal parameters $\bm{\hat{\theta}}$ we can evaluate the expectation value
|
||||
!bt
|
||||
\[
|
||||
\mathbb{E}(\bm{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\beta}=\bm{\beta}.
|
||||
\mathbb{E}(\bm{\hat{\theta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\theta}=\bm{\theta}.
|
||||
\]
|
||||
!et
|
||||
This means that the estimator of the regression parameters is unbiased.
|
||||
|
||||
We can also calculate the variance
|
||||
|
||||
The variance of the optimal value $\bm{\hat{\beta}}$ is
|
||||
The variance of the optimal value $\bm{\hat{\theta}}$ is
|
||||
!bt
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\bm{\hat{\beta}}) & = & \mathbb{E} \{ [\bm{\beta} - \mathbb{E}(\bm{\beta})] [\bm{\beta} - \mathbb{E}(\bm{\beta})]^{T} \}
|
||||
\mbox{Var}(\bm{\hat{\theta}}) & = & \mathbb{E} \{ [\bm{\theta} - \mathbb{E}(\bm{\theta})] [\bm{\theta} - \mathbb{E}(\bm{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\beta} \, \bm{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\theta} \, \bm{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\beta} \, \bm{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\theta} \, \bm{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\theta} \, \bm{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\beta} \, \bm{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\theta} \, \bm{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\beta} \bm{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\theta} \bm{\theta}^T
|
||||
\\
|
||||
& = & \bm{\beta} \, \bm{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
|
||||
& = & \bm{\theta} \, \bm{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
!et
|
||||
|
||||
where we have used that $\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \bm{\theta} \, \bm{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the
|
||||
variance of the estimate of the $j$-th regression coefficient:
|
||||
$\bm{\sigma}^2 (\bm{\beta}_j ) = \bm{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $. This may be used to
|
||||
$\bm{\sigma}^2 (\bm{\theta}_j ) = \bm{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $. This may be used to
|
||||
construct a confidence interval for the estimates.
|
||||
|
||||
|
||||
In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters $\bm{\beta}$ and their variance
|
||||
expectation values of the parameters $\bm{\theta}$ and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.
|
||||
|
||||
It is rather straightforward to show that
|
||||
!bt
|
||||
\[
|
||||
\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \bm{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\theta}^{\mathrm{OLS}}.
|
||||
\]
|
||||
!et
|
||||
We see clearly that
|
||||
$\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big] \not= \bm{\beta}^{\mathrm{OLS}}$ for any $\lambda > 0$. We say then that the ridge estimator is biased.
|
||||
$\mathbb{E} \big[ \bm{\theta}^{\mathrm{Ridge}} \big] \not= \bm{\theta}^{\mathrm{OLS}}$ for any $\lambda > 0$. We say then that the ridge estimator is biased.
|
||||
|
||||
We can also compute the variance as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\bm{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\]
|
||||
!et
|
||||
and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero.
|
||||
and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\theta}$ goes to zero.
|
||||
|
||||
With this, we can compute the difference
|
||||
|
||||
!bt
|
||||
\[
|
||||
\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\bm{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\]
|
||||
!et
|
||||
The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
|
||||
|
||||
|
||||
!split
|
||||
@@ -209,15 +209,15 @@ distribution with zero mean value and an undetermined variance
|
||||
$\sigma^2$.
|
||||
|
||||
We found above that the outputs $\bm{y}$ have a mean value given by
|
||||
$\bm{X}\hat{\bm{\beta}}$ and variance $\sigma^2$. Since the entries to
|
||||
$\bm{X}\hat{\bm{\theta}}$ and variance $\sigma^2$. Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value $\bm{X}\hat{\bm{\beta}}$. This means that a
|
||||
but now with mean value $\bm{X}\hat{\bm{\theta}}$. This means that a
|
||||
single output $y_i$ is given by the Gaussian distribution
|
||||
|
||||
!bt
|
||||
\[
|
||||
y_i\sim \mathcal{N}(\bm{X}_{i,*}\bm{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\bm{X}_{i,*}\bm{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -228,16 +228,16 @@ We assume now that the various $y_i$ values are stochastically distributed accor
|
||||
We define this distribution as
|
||||
!bt
|
||||
\[
|
||||
p(y_i, \bm{X}\vert\bm{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \bm{X}\vert\bm{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]},
|
||||
\]
|
||||
!et
|
||||
which reads as finding the likelihood of an event $y_i$ with the input variables $\bm{X}$ given the parameters (to be determined) $\bm{\beta}$.
|
||||
which reads as finding the likelihood of an event $y_i$ with the input variables $\bm{X}$ given the parameters (to be determined) $\bm{\theta}$.
|
||||
|
||||
Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\bm{y}$ as the product of the single events, that is we have
|
||||
|
||||
!bt
|
||||
\[
|
||||
p(\bm{y},\bm{X}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\beta}).
|
||||
p(\bm{y},\bm{X}\vert\bm{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\theta}).
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -252,11 +252,11 @@ In the more general case the various inputs should be replaced by the possible f
|
||||
We can now rewrite the above probability as
|
||||
!bt
|
||||
\[
|
||||
p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\bm{D}\vert\bm{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]
|
||||
!et
|
||||
|
||||
It is a conditional probability (see below) and reads as the likelihood of a domain of events $\bm{D}$ given a set of parameters $\bm{\beta}$.
|
||||
It is a conditional probability (see below) and reads as the likelihood of a domain of events $\bm{D}$ given a set of parameters $\bm{\theta}$.
|
||||
|
||||
!split
|
||||
===== Maximum Likelihood Estimation (MLE) =====
|
||||
@@ -269,7 +269,7 @@ data is the most probable.
|
||||
|
||||
|
||||
We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters $\beta$ by
|
||||
distribution and we will determine the optimal parameters $\theta$ by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.
|
||||
@@ -293,27 +293,27 @@ We could now define a new cost function to minimize, namely the negative logarit
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\bm{X}\vert\bm{\beta})},
|
||||
C(\bm{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\bm{X}\vert\bm{\theta})},
|
||||
\]
|
||||
!et
|
||||
which becomes
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\bm{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
\]
|
||||
!et
|
||||
|
||||
Taking the derivative of the *new* cost function with respect to the parameters $\beta$ we recognize our familiar OLS equation, namely
|
||||
Taking the derivative of the *new* cost function with respect to the parameters $\theta$ we recognize our familiar OLS equation, namely
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right) =0,
|
||||
\bm{X}^T\left(\bm{y}-\bm{X}\bm{\theta}\right) =0,
|
||||
\]
|
||||
!et
|
||||
which leads to the well-known OLS equation for the optimal paramters $\beta$
|
||||
which leads to the well-known OLS equation for the optimal paramters $\theta$
|
||||
!bt
|
||||
\[
|
||||
\hat{\bm{\beta}}^{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}!
|
||||
\hat{\bm{\theta}}^{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}!
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -601,13 +601,13 @@ $\tilde{p}(x)$.
|
||||
|
||||
Confidence intervals are used in statistics and represent a type of estimate
|
||||
computed from the observed data. This gives a range of values for an
|
||||
unknown parameter such as the parameters $\bm{\beta}$ from linear regression.
|
||||
unknown parameter such as the parameters $\bm{\theta}$ from linear regression.
|
||||
|
||||
With the OLS expressions for the parameters $\bm{\beta}$ we found
|
||||
$\mathbb{E}(\bm{\beta}) = \bm{\beta}$, which means that the estimator of the regression parameters is unbiased.
|
||||
With the OLS expressions for the parameters $\bm{\theta}$ we found
|
||||
$\mathbb{E}(\bm{\theta}) = \bm{\theta}$, which means that the estimator of the regression parameters is unbiased.
|
||||
|
||||
In the exercises this week we show that the variance of the estimate of the $j$-th regression coefficient is
|
||||
$\bm{\sigma}^2 (\bm{\beta}_j ) = \bm{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $.
|
||||
$\bm{\sigma}^2 (\bm{\theta}_j ) = \bm{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $.
|
||||
|
||||
This quantity can be used to
|
||||
construct a confidence interval for the estimates.
|
||||
@@ -616,14 +616,14 @@ construct a confidence interval for the estimates.
|
||||
!split
|
||||
===== Standard Approach based on the Normal Distribution =====
|
||||
|
||||
We will assume that the parameters $\beta$ follow a normal
|
||||
We will assume that the parameters $\theta$ follow a normal
|
||||
distribution. We can then define the confidence interval. Here we will be using as
|
||||
shorthands $\mu_{\beta}$ for the above mean value and $\sigma_{\beta}$
|
||||
shorthands $\mu_{\theta}$ for the above mean value and $\sigma_{\theta}$
|
||||
for the standard deviation. We have then a confidence interval
|
||||
|
||||
!bt
|
||||
\[
|
||||
\left(\mu_{\beta}\pm \frac{z\sigma_{\beta}}{\sqrt{n}}\right),
|
||||
\left(\mu_{\theta}\pm \frac{z\sigma_{\theta}}{\sqrt{n}}\right),
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -642,11 +642,11 @@ Bootstrap method, why it works and various theorems related to it.
|
||||
!split
|
||||
===== Resampling methods: Bootstrap background =====
|
||||
|
||||
Since $\widehat{\beta} = \widehat{\beta}(\bm{X})$ is a function of random variables,
|
||||
$\widehat{\beta}$ itself must be a random variable. Thus it has
|
||||
Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables,
|
||||
$\widehat{\theta}$ itself must be a random variable. Thus it has
|
||||
a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to
|
||||
estimate $p(\bm{t})$ by the relative frequency of
|
||||
$\widehat{\beta}$. You can think of this as using a histogram
|
||||
$\widehat{\theta}$. You can think of this as using a histogram
|
||||
in the place of $p(\bm{t})$. If the relative frequency closely
|
||||
resembles $p(\vec{t})$, then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of $p(\bm{t})$ using point
|
||||
@@ -656,17 +656,17 @@ estimators.
|
||||
!split
|
||||
===== Resampling methods: More Bootstrap background =====
|
||||
|
||||
In the case that $\widehat{\beta}$ has
|
||||
In the case that $\widehat{\theta}$ has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of $X_i$, $p(x)$, had been known, then it would have
|
||||
been straightforward to do this by:
|
||||
o Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$.
|
||||
o Then using these numbers, we could compute a replica of $\widehat{\beta}$ called $\widehat{\beta}^*$.
|
||||
o Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$.
|
||||
|
||||
By repeated use of the above two points, many
|
||||
estimates of $\widehat{\beta}$ can be obtained. The
|
||||
idea is to use the relative frequency of $\widehat{\beta}^*$
|
||||
estimates of $\widehat{\theta}$ can be obtained. The
|
||||
idea is to use the relative frequency of $\widehat{\theta}^*$
|
||||
(think of a histogram) as an estimate of $p(\bm{t})$.
|
||||
|
||||
!split
|
||||
@@ -692,18 +692,18 @@ The independent bootstrap works like this:
|
||||
|
||||
o Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$.
|
||||
o Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$.
|
||||
o Using the vector $\bm{x}^*$ compute $\widehat{\beta}^*$ by evaluating $\widehat \beta$ under the observations $\bm{x}^*$.
|
||||
o Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$.
|
||||
o Repeat this process $k$ times.
|
||||
|
||||
When you are done, you can draw a histogram of the relative frequency
|
||||
of $\widehat \beta^*$. This is your estimate of the probability
|
||||
of $\widehat \theta^*$. This is your estimate of the probability
|
||||
distribution $p(t)$. Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of $\widehat{\beta}^*$. Instead
|
||||
histogram of the relative frequency of $\widehat{\theta}^*$. Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of $\widehat
|
||||
\beta$, apply the etsimator $\widehat \sigma^2$ to the values
|
||||
$\widehat \beta^*$.
|
||||
\theta$, apply the etsimator $\widehat \sigma^2$ to the values
|
||||
$\widehat \theta^*$.
|
||||
|
||||
|
||||
!split
|
||||
@@ -791,13 +791,13 @@ where $\epsilon$ is normally distributed with mean zero and standard deviation $
|
||||
|
||||
In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function $f$ in terms of the parameters
|
||||
$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model,
|
||||
that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$.
|
||||
$\bm{\theta}$ and the design matrix $\bm{X}$ which embody our model,
|
||||
that is $\bm{\tilde{y}}=\bm{X}\bm{\theta}$.
|
||||
|
||||
Thereafter we found the parameters $\bm{\beta}$ by optimizing the means squared error via the so-called cost function
|
||||
Thereafter we found the parameters $\bm{\theta}$ by optimizing the means squared error via the so-called cost function
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
|
||||
C(\bm{X},\bm{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
|
||||
\]
|
||||
!et
|
||||
|
||||
|
||||
Reference in New Issue
Block a user