beta--> theta
This commit is contained in:
Morten Hjorth-Jensen
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
@@ -391,7 +391,7 @@ document.write(`
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#assumptions-made">Assumptions made</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance">Expectation value and variance</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-theta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#deriving-ols-from-a-probability-distribution">Deriving OLS from a probability distribution</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#independent-and-identically-distributed-iid">Independent and Identically Distributed (iid)</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
@@ -474,7 +474,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
<span class="math notranslate nohighlight">\(\beta\)</span>.</p>
|
||||
<span class="math notranslate nohighlight">\(\theta\)</span>.</p>
|
||||
<p>On of the advantages of doing linear regression is that we actually end up with
|
||||
analytical expressions for several statistical quantities.<br />
|
||||
Standard least squares and Ridge regression allow us to
|
||||
@@ -494,7 +494,7 @@ independent, i.e.:</p>
|
||||
<p>The randomness of <span class="math notranslate nohighlight">\(\varepsilon_i\)</span> implies that
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is also a random variable. In particular,
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is normally distributed, because <span class="math notranslate nohighlight">\(\varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\)</span> is a
|
||||
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\theta}\)</span> is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> we need to calculate its first two moments.</p>
|
||||
<p>Recall that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is a matrix of dimensionality <span class="math notranslate nohighlight">\(n\times p\)</span>. The
|
||||
@@ -514,7 +514,7 @@ which describe our data</p>
|
||||
function <span class="math notranslate nohighlight">\(f\)</span> is approximated by <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> where we want to minimize <span class="math notranslate nohighlight">\((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\)</span>, our MSE, with</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
|
||||
\]</div>
|
||||
</section>
|
||||
<section id="expectation-value-and-variance">
|
||||
@@ -524,8 +524,8 @@ function <span class="math notranslate nohighlight">\(f\)</span> is approximated
|
||||
\[
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
\]</div>
|
||||
<p>while
|
||||
@@ -535,83 +535,83 @@ its variance is</p>
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
\end{split}\]</div>
|
||||
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
|
||||
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\beta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
|
||||
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
|
||||
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\theta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
|
||||
</section>
|
||||
<section id="expectation-value-and-variance-for-boldsymbol-beta">
|
||||
<h2>Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span><a class="headerlink" href="#expectation-value-and-variance-for-boldsymbol-beta" title="Link to this heading">#</a></h2>
|
||||
<p>With the OLS expressions for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}}\)</span> we can evaluate the expectation value</p>
|
||||
<section id="expectation-value-and-variance-for-boldsymbol-theta">
|
||||
<h2>Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#expectation-value-and-variance-for-boldsymbol-theta" title="Link to this heading">#</a></h2>
|
||||
<p>With the OLS expressions for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\theta}}\)</span> we can evaluate the expectation value</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
|
||||
\mathbb{E}(\boldsymbol{\hat{\theta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
|
||||
\]</div>
|
||||
<p>This means that the estimator of the regression parameters is unbiased.</p>
|
||||
<p>We can also calculate the variance</p>
|
||||
<p>The variance of the optimal value <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}}\)</span> is</p>
|
||||
<p>The variance of the optimal value <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\theta}}\)</span> is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
|
||||
\mbox{Var}(\boldsymbol{\hat{\theta}}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
|
||||
\\
|
||||
& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
\end{split}\]</div>
|
||||
<p>where we have used that <span class="math notranslate nohighlight">\(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1}\)</span>, one obtains an estimate of the
|
||||
variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient:
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
|
||||
construct a confidence interval for the estimates.</p>
|
||||
<p>In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and their variance
|
||||
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.</p>
|
||||
<p>It is rather straightforward to show that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
|
||||
\]</div>
|
||||
<p>We see clearly that
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda > 0\)</span>. We say then that the ridge estimator is biased.</p>
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda > 0\)</span>. We say then that the ridge estimator is biased.</p>
|
||||
<p>We can also compute the variance as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\]</div>
|
||||
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> goes to zero.</p>
|
||||
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> goes to zero.</p>
|
||||
<p>With this, we can compute the difference</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\]</div>
|
||||
<p>The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda > 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
|
||||
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda > 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
|
||||
</section>
|
||||
<section id="deriving-ols-from-a-probability-distribution">
|
||||
<h2>Deriving OLS from a probability distribution<a class="headerlink" href="#deriving-ols-from-a-probability-distribution" title="Link to this heading">#</a></h2>
|
||||
@@ -621,14 +621,14 @@ that our output is determined by a given continuous function
|
||||
distribution with zero mean value and an undetermined variance
|
||||
<span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
|
||||
<p>We found above that the outputs <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> have a mean value given by
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span>. This means that a
|
||||
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span>. This means that a
|
||||
single output <span class="math notranslate nohighlight">\(y_i\)</span> is given by the Gaussian distribution</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]</div>
|
||||
</section>
|
||||
<section id="independent-and-identically-distributed-iid">
|
||||
@@ -637,13 +637,13 @@ y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\
|
||||
We define this distribution as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
|
||||
\]</div>
|
||||
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
|
||||
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
|
||||
\]</div>
|
||||
<p>We will write this in a more compact form reserving <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> for the domain of events, including the ouputs (targets) and the inputs. That is
|
||||
in case we have a simple one-dimensional input and output case</p>
|
||||
@@ -655,9 +655,9 @@ in case we have a simple one-dimensional input and output case</p>
|
||||
We can now rewrite the above probability as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]</div>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
|
||||
</section>
|
||||
<section id="maximum-likelihood-estimation-mle">
|
||||
<h2>Maximum Likelihood Estimation (MLE)<a class="headerlink" href="#maximum-likelihood-estimation-mle" title="Link to this heading">#</a></h2>
|
||||
@@ -667,7 +667,7 @@ given some observed data. This is achieved by maximizing a likelihood
|
||||
function so that, under the assumed statistical model, the observed
|
||||
data is the most probable.</p>
|
||||
<p>We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> by
|
||||
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\theta\)</span> by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.</p>
|
||||
@@ -684,22 +684,22 @@ is equivalent to the maximization/minimization of the function itself.</p>
|
||||
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
|
||||
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
|
||||
\]</div>
|
||||
<p>which becomes</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
\]</div>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\beta\)</span> we recognize our familiar OLS equation, namely</p>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\theta\)</span> we recognize our familiar OLS equation, namely</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
|
||||
\]</div>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\beta\)</span></p>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\theta\)</span></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\]</div>
|
||||
<p>Next week we will make a similar analysis for Ridge and Lasso regression</p>
|
||||
</section>
|
||||
@@ -926,23 +926,23 @@ finite <span class="math notranslate nohighlight">\(m\)</span>, it is not always
|
||||
<h2>Confidence Intervals<a class="headerlink" href="#confidence-intervals" title="Link to this heading">#</a></h2>
|
||||
<p>Confidence intervals are used in statistics and represent a type of estimate
|
||||
computed from the observed data. This gives a range of values for an
|
||||
unknown parameter such as the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> from linear regression.</p>
|
||||
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we found
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta}\)</span>, which means that the estimator of the regression parameters is unbiased.</p>
|
||||
unknown parameter such as the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> from linear regression.</p>
|
||||
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we found
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E}(\boldsymbol{\theta}) = \boldsymbol{\theta}\)</span>, which means that the estimator of the regression parameters is unbiased.</p>
|
||||
<p>In the exercises this week we show that the variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient is
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>.</p>
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>.</p>
|
||||
<p>This quantity can be used to
|
||||
construct a confidence interval for the estimates.</p>
|
||||
</section>
|
||||
<section id="standard-approach-based-on-the-normal-distribution">
|
||||
<h2>Standard Approach based on the Normal Distribution<a class="headerlink" href="#standard-approach-based-on-the-normal-distribution" title="Link to this heading">#</a></h2>
|
||||
<p>We will assume that the parameters <span class="math notranslate nohighlight">\(\beta\)</span> follow a normal
|
||||
<p>We will assume that the parameters <span class="math notranslate nohighlight">\(\theta\)</span> follow a normal
|
||||
distribution. We can then define the confidence interval. Here we will be using as
|
||||
shorthands <span class="math notranslate nohighlight">\(\mu_{\beta}\)</span> for the above mean value and <span class="math notranslate nohighlight">\(\sigma_{\beta}\)</span>
|
||||
shorthands <span class="math notranslate nohighlight">\(\mu_{\theta}\)</span> for the above mean value and <span class="math notranslate nohighlight">\(\sigma_{\theta}\)</span>
|
||||
for the standard deviation. We have then a confidence interval</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\left(\mu_{\beta}\pm \frac{z\sigma_{\beta}}{\sqrt{n}}\right),
|
||||
\left(\mu_{\theta}\pm \frac{z\sigma_{\theta}}{\sqrt{n}}\right),
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(z\)</span> defines the level of certainty (or confidence). For a normal
|
||||
distribution typical parameters are <span class="math notranslate nohighlight">\(z=2.576\)</span> which corresponds to a
|
||||
@@ -956,11 +956,11 @@ Bootstrap method, why it works and various theorems related to it.</p>
|
||||
</section>
|
||||
<section id="resampling-methods-bootstrap-background">
|
||||
<h2>Resampling methods: Bootstrap background<a class="headerlink" href="#resampling-methods-bootstrap-background" title="Link to this heading">#</a></h2>
|
||||
<p>Since <span class="math notranslate nohighlight">\(\widehat{\beta} = \widehat{\beta}(\boldsymbol{X})\)</span> is a function of random variables,
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> itself must be a random variable. Thus it has
|
||||
<p>Since <span class="math notranslate nohighlight">\(\widehat{\theta} = \widehat{\theta}(\boldsymbol{X})\)</span> is a function of random variables,
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> itself must be a random variable. Thus it has
|
||||
a pdf, call this function <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. The aim of the bootstrap is to
|
||||
estimate <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> by the relative frequency of
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>. You can think of this as using a histogram
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>. You can think of this as using a histogram
|
||||
in the place of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. If the relative frequency closely
|
||||
resembles <span class="math notranslate nohighlight">\(p(\vec{t})\)</span>, then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> using point
|
||||
@@ -968,18 +968,18 @@ estimators.</p>
|
||||
</section>
|
||||
<section id="resampling-methods-more-bootstrap-background">
|
||||
<h2>Resampling methods: More Bootstrap background<a class="headerlink" href="#resampling-methods-more-bootstrap-background" title="Link to this heading">#</a></h2>
|
||||
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> has
|
||||
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of <span class="math notranslate nohighlight">\(X_i\)</span>, <span class="math notranslate nohighlight">\(p(x)\)</span>, had been known, then it would have
|
||||
been straightforward to do this by:</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Drawing lots of numbers from <span class="math notranslate nohighlight">\(p(x)\)</span>, suppose we call one such set of numbers <span class="math notranslate nohighlight">\((X_1^*, X_2^*, \cdots, X_n^*)\)</span>.</p></li>
|
||||
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>.</p></li>
|
||||
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>.</p></li>
|
||||
</ol>
|
||||
<p>By repeated use of the above two points, many
|
||||
estimates of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> can be obtained. The
|
||||
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>
|
||||
estimates of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> can be obtained. The
|
||||
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>
|
||||
(think of a histogram) as an estimate of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>.</p>
|
||||
</section>
|
||||
<section id="resampling-methods-bootstrap-approach">
|
||||
@@ -1000,18 +1000,18 @@ result in some asymptotic sense? The answer is yes.</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Draw with replacement <span class="math notranslate nohighlight">\(n\)</span> numbers for the observed variables <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,x_n)\)</span>.</p></li>
|
||||
<li><p>Define a vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> containing the values which were drawn from <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>.</p></li>
|
||||
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \beta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
|
||||
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \theta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
|
||||
<li><p>Repeat this process <span class="math notranslate nohighlight">\(k\)</span> times.</p></li>
|
||||
</ol>
|
||||
<p>When you are done, you can draw a histogram of the relative frequency
|
||||
of <span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>. This is your estimate of the probability
|
||||
of <span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>. This is your estimate of the probability
|
||||
distribution <span class="math notranslate nohighlight">\(p(t)\)</span>. Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>. Instead
|
||||
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>. Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of <span class="math notranslate nohighlight">\(\widehat
|
||||
\beta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
|
||||
<span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>.</p>
|
||||
\theta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
|
||||
<span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>.</p>
|
||||
</section>
|
||||
<section id="code-example-for-the-bootstrap-method">
|
||||
<h2>Code example for the Bootstrap method<a class="headerlink" href="#code-example-for-the-bootstrap-method" title="Link to this heading">#</a></h2>
|
||||
@@ -1096,12 +1096,12 @@ tasks. Consider a dataset <span class="math notranslate nohighlight">\(\mathcal{
|
||||
<p>where <span class="math notranslate nohighlight">\(\epsilon\)</span> is normally distributed with mean zero and standard deviation <span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
|
||||
<p>In our derivation of the ordinary least squares method we defined then
|
||||
an approximation to the function <span class="math notranslate nohighlight">\(f\)</span> in terms of the parameters
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
|
||||
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>Thereafter we found the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> by optimizing the means squared error via the so-called cost function</p>
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
|
||||
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta}\)</span>.</p>
|
||||
<p>Thereafter we found the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> by optimizing the means squared error via the so-called cost function</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
|
||||
\]</div>
|
||||
<p>We can rewrite this as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -1734,7 +1734,7 @@ the jupyter-notebook from week 37 (September 12-16).</p>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#assumptions-made">Assumptions made</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance">Expectation value and variance</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#expectation-value-and-variance-for-boldsymbol-theta">Expectation value and variance for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#deriving-ols-from-a-probability-distribution">Deriving OLS from a probability distribution</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#independent-and-identically-distributed-iid">Independent and Identically Distributed (iid)</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maximum-likelihood-estimation-mle">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+197
-197
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user