lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-08-18 10:47:43 +02:00
parent d4db3104e6
commit 35c45696d6
37 changed files with 5815 additions and 5344 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/E1.html
+1 -1
View File
@@ -32,7 +32,7 @@
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
doc/LectureNotes/_build/html/E2.html
+20 -10
View File
@@ -469,37 +469,37 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
</section>
<section id="exercise-2-deriving-the-expression-for-ols">
<h2>Exercise 2 - Deriving the expression for OLS<a class="headerlink" href="#exercise-2-deriving-the-expression-for-ols" title="Link to this heading">#</a></h2>
<p>The ordinary least squares method finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes the squared error between our model <span class="math notranslate nohighlight">\(\boldsymbol{X\beta}\)</span> and the true values <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</p>
<p>To find the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes this error, we take the derivative of the squared error expression with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, and set it equal to 0.</p>
<p><strong>a)</strong> Very briefly explain why the approach above finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes this error.</p>
<p>The ordinary least squares method finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> which minimizes the squared error between our model <span class="math notranslate nohighlight">\(\boldsymbol{X\theta}\)</span> and the true values <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</p>
<p>To find the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> which minimizes this error, we take the derivative of the squared error expression with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>, and set it equal to 0.</p>
<p><strong>a)</strong> Very briefly explain why the approach above finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> which minimizes this error.</p>
<p>We typically write the squared error as</p>
<div class="math notranslate nohighlight">
\[
\vert\vert\boldsymbol{y} - \boldsymbol{X\beta}\vert\vert^2
\vert\vert\boldsymbol{y} - \boldsymbol{X\theta}\vert\vert^2
\]</div>
<p>which we can rewrite in matrix-vector form as</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)
\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)
\]</div>
<p><strong>b)</strong> If <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is invertible, what is the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>? (<strong>Hint:</strong> Don’t compute any derivatives, but solve <span class="math notranslate nohighlight">\(\boldsymbol{X\beta}=\boldsymbol{y}\)</span> for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>)</p>
<p><strong>b)</strong> If <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is invertible, what is the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>? (<strong>Hint:</strong> Don’t compute any derivatives, but solve <span class="math notranslate nohighlight">\(\boldsymbol{X\theta}=\boldsymbol{y}\)</span> for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>)</p>
<p><strong>c)</strong> Show that</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)}{\partial \boldsymbol{s}} = -2\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\boldsymbol{A},
\]</div>
<p><strong>d)</strong> Using the expression from <strong>c)</strong>, but substituting back in <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>, find the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> in the case that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is not invertible, but <span class="math notranslate nohighlight">\(\boldsymbol{X^T X}\)</span> is, which is most often the case.</p>
<p><strong>d)</strong> Using the expression from <strong>c)</strong>, but substituting back in <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>, <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>, find the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> in the case that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is not invertible, but <span class="math notranslate nohighlight">\(\boldsymbol{X^T X}\)</span> is, which is most often the case.</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\hat{\beta}_{OLS}} = ...
\boldsymbol{\hat{\theta}_{OLS}} = ...
\]</div>
</section>
<section id="exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression">
<h2>Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression<a class="headerlink" href="#exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression" title="Link to this heading">#</a></h2>
<p>With the expression for <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}_{OLS}}\)</span>, you now have what you need to implement OLS regression with your input data and target data <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. But before you can do that, you need to set up you input data as a feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>With the expression for <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\theta}_{OLS}}\)</span>, you now have what you need to implement OLS regression with your input data and target data <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. But before you can do that, you need to set up you input data as a feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.</p>
<p>We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.</p>
<p>We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter <span class="math notranslate nohighlight">\(\beta_0\)</span> from the OLS method into the optimal intercept value (which will be <span class="math notranslate nohighlight">\(\beta_0\)</span>). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.</p>
<p>We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter <span class="math notranslate nohighlight">\(\theta_0\)</span> from the OLS method into the optimal intercept value (which will be <span class="math notranslate nohighlight">\(\theta_0\)</span>). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
@@ -594,6 +594,11 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Ellipsis
</pre></div>
</div>
</div>
</div>
<p><strong>e)</strong> Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of <a class="reference external" href="https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf">Hastie et al</a>. Feel free to read the discussions leading to figure 2.11 of Hastie et al.</p>
<div class="cell docutils container">
@@ -602,6 +607,11 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Ellipsis
</pre></div>
</div>
</div>
</div>
<p><strong>f)</strong> Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?</p>
</section>
doc/LectureNotes/_build/html/_sources/E2.ipynb
+46 -24
View File
@@ -146,9 +146,9 @@
"id": "f3f771de",
"metadata": {},
"source": [
"The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n",
"The ordinary least squares method finds the parameters $\\boldsymbol{\\theta}$ which minimizes the squared error between our model $\\boldsymbol{X\\theta}$ and the true values $\\boldsymbol{y}$.\n",
"\n",
"To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0."
"To find the parameters $\\boldsymbol{\\theta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\theta}$, and set it equal to 0."
]
},
{
@@ -156,7 +156,7 @@
"id": "49690237",
"metadata": {},
"source": [
"**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error."
"**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\theta}$ which minimizes this error."
]
},
{
@@ -167,13 +167,13 @@
"We typically write the squared error as\n",
"\n",
"$$\n",
"\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n",
"\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\theta}\\vert\\vert^2\n",
"$$\n",
"\n",
"which we can rewrite in matrix-vector form as\n",
"\n",
"$$\n",
"\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n",
"\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\n",
"$$"
]
},
@@ -182,7 +182,7 @@
"id": "8fbecf74",
"metadata": {},
"source": [
"**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)"
"**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\theta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\theta}=\\boldsymbol{y}$ for $\\boldsymbol{\\theta}$)"
]
},
{
@@ -202,10 +202,10 @@
"id": "869fca4d",
"metadata": {},
"source": [
"**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n",
"**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\theta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\theta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n",
"\n",
"$$\n",
"\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n",
"\\boldsymbol{\\hat{\\theta}_{OLS}} = ...\n",
"$$"
]
},
@@ -222,18 +222,18 @@
"id": "5dc179f7",
"metadata": {},
"source": [
"With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n",
"With the expression for $\\boldsymbol{\\hat{\\theta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n",
"\n",
"In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.\n",
"\n",
"We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.\n",
"\n",
"We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\beta_0$ from the OLS method into the optimal intercept value (which will be $\\beta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix."
"We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\theta_0$ from the OLS method into the optimal intercept value (which will be $\\theta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix."
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"id": "e5ff2a69",
"metadata": {},
"outputs": [],
@@ -243,7 +243,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"id": "a3cf2792",
"metadata": {},
"outputs": [],
@@ -264,7 +264,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 3,
"id": "5ad87a65",
"metadata": {},
"outputs": [],
@@ -285,7 +285,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 4,
"id": "8f3f68aa",
"metadata": {},
"outputs": [],
@@ -316,7 +316,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 5,
"id": "d7476c84",
"metadata": {},
"outputs": [],
@@ -336,7 +336,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 6,
"id": "91496e40",
"metadata": {},
"outputs": [],
@@ -362,7 +362,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"id": "034f502c",
"metadata": {},
"outputs": [],
@@ -380,7 +380,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"id": "29171358",
"metadata": {},
"outputs": [],
@@ -400,10 +400,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 9,
"id": "1e346f4c",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"Ellipsis"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"..."
]
@@ -418,10 +429,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 10,
"id": "ceb57457",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"Ellipsis"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"..."
]
@@ -495,7 +517,7 @@
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
@@ -509,7 +531,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.0"
"version": "3.9.15"
}
},
"nbformat": 4,
doc/LectureNotes/_build/html/_sources/chapter1.ipynb
+433 -466
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/_sources/chapter2.ipynb
+539 -539
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/_sources/chapter3.ipynb
+340 -340
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/_sources/chapter4.ipynb
+97 -97
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "f63b8c1a",
"id": "225f0ca7",
"metadata": {
"editable": true
},
@@ -13,7 +13,7 @@
},
{
"cell_type": "markdown",
"id": "9a284cf5",
"id": "a48a2920",
"metadata": {
"editable": true
},
@@ -23,7 +23,7 @@
},
{
"cell_type": "markdown",
"id": "d78b73e2",
"id": "90b6e822",
"metadata": {
"editable": true
},
@@ -38,7 +38,7 @@
"analytical expressions for standard ordinary Least Squares or Ridge\n",
"regression (in terms of matrices to invert) for several quantities,\n",
"ranging from the variance and thereby the confidence intervals of the\n",
"optimal parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
"optimal parameters $\\hat{\\theta}$ to the mean squared error. If we can invert\n",
"the product of the design matrices, linear regression gives then a\n",
"simple recipe for fitting our data.\n",
"\n",
@@ -61,7 +61,7 @@
"Logistic regression will also serve as our stepping stone towards\n",
"neural network algorithms and supervised deep learning. For logistic\n",
"learning, the minimization of the cost function leads to a non-linear\n",
"equation in the parameters $\\hat{\\beta}$. The optimization of the\n",
"equation in the parameters $\\hat{\\theta}$. The optimization of the\n",
"problem calls therefore for minimization algorithms. This forms the\n",
"bottle neck of all machine learning algorithms, namely how to find\n",
"reliable minima of a multi-variable function. This leads us to the\n",
@@ -75,7 +75,7 @@
},
{
"cell_type": "markdown",
"id": "d0c829b4",
"id": "08ebdf8e",
"metadata": {
"editable": true
},
@@ -100,7 +100,7 @@
},
{
"cell_type": "markdown",
"id": "f1f4d5f2",
"id": "f08eca95",
"metadata": {
"editable": true
},
@@ -112,7 +112,7 @@
},
{
"cell_type": "markdown",
"id": "8e2f836b",
"id": "d82a2a29",
"metadata": {
"editable": true
},
@@ -128,7 +128,7 @@
},
{
"cell_type": "markdown",
"id": "6b58f1ba",
"id": "467f1d23",
"metadata": {
"editable": true
},
@@ -138,7 +138,7 @@
"\n",
"$$\n",
"\\begin{equation}\n",
"\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n",
"\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon},\n",
"\\label{_auto1} \\tag{1}\n",
"\\end{equation}\n",
"$$"
@@ -146,13 +146,13 @@
},
{
"cell_type": "markdown",
"id": "54e3604b",
"id": "0f2560d0",
"metadata": {
"editable": true
},
"source": [
"where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
"$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n",
"$n\\times p$ design matrix and $\\boldsymbol{\\theta}$ represents our estimators/predictors.\n",
"\n",
"The main problem with our function is that it takes values on the\n",
"entire real axis. In the case of logistic regression, however, the\n",
@@ -175,7 +175,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "cd1dd1c5",
"id": "3916f494",
"metadata": {
"collapsed": false,
"editable": true
@@ -242,7 +242,7 @@
},
{
"cell_type": "markdown",
"id": "2e06ee82",
"id": "df52e736",
"metadata": {
"editable": true
},
@@ -253,7 +253,7 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "a774f001",
"id": "07b5c0a7",
"metadata": {
"collapsed": false,
"editable": true
@@ -272,7 +272,7 @@
},
{
"cell_type": "markdown",
"id": "e102fa56",
"id": "25ce17fc",
"metadata": {
"editable": true
},
@@ -283,19 +283,19 @@
},
{
"cell_type": "markdown",
"id": "75d995ba",
"id": "818e2f55",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n",
"f(y_i\\vert x_i)=\\theta_0+\\theta_1 x_i.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "e2cacca5",
"id": "b2616b20",
"metadata": {
"editable": true
},
@@ -314,7 +314,7 @@
},
{
"cell_type": "markdown",
"id": "af9ac586",
"id": "ab85db3c",
"metadata": {
"editable": true
},
@@ -336,7 +336,7 @@
},
{
"cell_type": "markdown",
"id": "80aa3cbe",
"id": "efbd4772",
"metadata": {
"editable": true
},
@@ -348,7 +348,7 @@
},
{
"cell_type": "markdown",
"id": "28d49915",
"id": "1b9231ea",
"metadata": {
"editable": true
},
@@ -358,7 +358,7 @@
},
{
"cell_type": "markdown",
"id": "eeff6cd9",
"id": "ad6acb90",
"metadata": {
"editable": true
},
@@ -371,7 +371,7 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "5ecec05e",
"id": "bf66d0dd",
"metadata": {
"collapsed": false,
"editable": true
@@ -436,56 +436,56 @@
},
{
"cell_type": "markdown",
"id": "5802efe3",
"id": "363760d3",
"metadata": {
"editable": true
},
"source": [
"We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
"We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\theta$ in our fitting of the Sigmoid function, that is we define probabilities"
]
},
{
"cell_type": "markdown",
"id": "f85d14ea",
"id": "eefe7b8d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\begin{align*}\n",
"p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
"p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
"p(y_i=1|x_i,\\boldsymbol{\\theta}) &= \\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}},\\nonumber\\\\\n",
"p(y_i=0|x_i,\\boldsymbol{\\theta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\theta}),\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "6c4e0334",
"id": "102fb347",
"metadata": {
"editable": true
},
"source": [
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"where $\\boldsymbol{\\theta}$ are the weights we wish to extract from data, in our case $\\theta_0$ and $\\theta_1$. \n",
"\n",
"Note that we used"
]
},
{
"cell_type": "markdown",
"id": "df7facc9",
"id": "0b230504",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
"p(y_i=0\\vert x_i, \\boldsymbol{\\theta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\theta}).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "a920e6d8",
"id": "669b4c9f",
"metadata": {
"editable": true
},
@@ -500,21 +500,21 @@
},
{
"cell_type": "markdown",
"id": "a780bfe1",
"id": "5dc36460",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\begin{align*}\n",
"P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
"P(\\mathcal{D}|\\boldsymbol{\\theta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\theta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]^{1-y_i}\\nonumber \\\\\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "9f810a44",
"id": "0874a9c6",
"metadata": {
"editable": true
},
@@ -524,19 +524,19 @@
},
{
"cell_type": "markdown",
"id": "3fac4ef4",
"id": "212397d3",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\theta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "75aa2c14",
"id": "f99396bf",
"metadata": {
"editable": true
},
@@ -546,42 +546,42 @@
},
{
"cell_type": "markdown",
"id": "780f2038",
"id": "fd5e7b5f",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "c8c940aa",
"id": "80dbca63",
"metadata": {
"editable": true
},
"source": [
"The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
"The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\theta$.\n",
"Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
]
},
{
"cell_type": "markdown",
"id": "9d4a527b",
"id": "f5d6c58f",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\theta})=-\\sum_{i=1}^n \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "c1706606",
"id": "57e33c81",
"metadata": {
"editable": true
},
@@ -589,28 +589,28 @@
"This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
"in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
"\n",
"The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n",
"The cross entropy is a convex function of the weights $\\boldsymbol{\\theta}$ and,\n",
"therefore, any local minimizer is a global minimizer. \n",
"\n",
"Minimizing this\n",
"cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
"cost function with respect to the two parameters $\\theta_0$ and $\\theta_1$ we obtain"
]
},
{
"cell_type": "markdown",
"id": "fe8fb387",
"id": "1833cc5d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "bb564e81",
"id": "ea34e8e6",
"metadata": {
"editable": true
},
@@ -620,67 +620,67 @@
},
{
"cell_type": "markdown",
"id": "700c5443",
"id": "f1bc373e",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "f442c956",
"id": "617c6d86",
"metadata": {
"editable": true
},
"source": [
"Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n",
"$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n",
"vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n",
"vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\theta})$. We can rewrite in a more compact form the first\n",
"derivative of cost function as"
]
},
{
"cell_type": "markdown",
"id": "83937f6f",
"id": "e79426c6",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "71819c1d",
"id": "b1dd9451",
"metadata": {
"editable": true
},
"source": [
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
"$p(y_i\\vert x_i,\\boldsymbol{\\theta})(1-p(y_i\\vert x_i,\\boldsymbol{\\theta})$, we can obtain a compact expression of the second derivative as"
]
},
{
"cell_type": "markdown",
"id": "23dfd975",
"id": "ac6500cc",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}\\partial \\boldsymbol{\\theta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "6ca9f74b",
"id": "12f4d5d0",
"metadata": {
"editable": true
},
@@ -690,41 +690,41 @@
},
{
"cell_type": "markdown",
"id": "ddb3d93a",
"id": "b4733616",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
"\\log{ \\frac{p(\\boldsymbol{\\theta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\theta}\\boldsymbol{x})}} = \\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "0cfae560",
"id": "eec7d85e",
"metadata": {
"editable": true
},
"source": [
"Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
"Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\theta}=[\\theta_0, \\theta_1, \\dots, \\theta_p]$ leading to"
]
},
{
"cell_type": "markdown",
"id": "7c427fc3",
"id": "152e6fa3",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
"p(\\boldsymbol{\\theta}\\boldsymbol{x})=\\frac{ \\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}{1+\\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "6a6814bf",
"id": "b8168ae0",
"metadata": {
"editable": true
},
@@ -736,19 +736,19 @@
},
{
"cell_type": "markdown",
"id": "85282137",
"id": "3af519ee",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n",
"\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\theta_{10}+\\theta_{11}x_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "4864f76d",
"id": "fa02c958",
"metadata": {
"editable": true
},
@@ -758,19 +758,19 @@
},
{
"cell_type": "markdown",
"id": "390e9a55",
"id": "c550e015",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
"\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\theta_{20}+\\theta_{21}x_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "c3f39c44",
"id": "8398ec7a",
"metadata": {
"editable": true
},
@@ -780,19 +780,19 @@
},
{
"cell_type": "markdown",
"id": "92f32a03",
"id": "6be24150",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
"\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\theta_{(K-1)0}+\\theta_{(K-1)1}x_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "677a5c4d",
"id": "2153c8e3",
"metadata": {
"editable": true
},
@@ -810,25 +810,25 @@
"multinomial logistic regression and linear discriminant analysis, the\n",
"input to the function is the result of $K$ distinct linear functions,\n",
"and the predicted probability for the $k$-th class given a sample\n",
"vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n",
"vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\theta}$ is (with two\n",
"predictors):"
]
},
{
"cell_type": "markdown",
"id": "fa70e3bf",
"id": "ab540777",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
"p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\theta_{k0}+\\theta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "4c59400c",
"id": "b7bcd7e8",
"metadata": {
"editable": true
},
@@ -838,19 +838,19 @@
},
{
"cell_type": "markdown",
"id": "f4073652",
"id": "c9605782",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
"p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "810e872a",
"id": "856dd896",
"metadata": {
"editable": true
},
@@ -867,7 +867,7 @@
},
{
"cell_type": "markdown",
"id": "f4ebc1da",
"id": "73029fad",
"metadata": {
"editable": true
},
@@ -882,7 +882,7 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "b7a8cbac",
"id": "d81a44e3",
"metadata": {
"collapsed": false,
"editable": true
@@ -918,7 +918,7 @@
},
{
"cell_type": "markdown",
"id": "10f74b93",
"id": "83a2e903",
"metadata": {
"editable": true
},
@@ -930,7 +930,7 @@
{
"cell_type": "code",
"execution_count": 5,
"id": "d0b8025d",
"id": "976fc321",
"metadata": {
"collapsed": false,
"editable": true
@@ -975,7 +975,7 @@
},
{
"cell_type": "markdown",
"id": "9d84a2aa",
"id": "ea10d953",
"metadata": {
"editable": true
},
@@ -998,7 +998,7 @@
{
"cell_type": "code",
"execution_count": 6,
"id": "6144ea0a",
"id": "42204f9c",
"metadata": {
"collapsed": false,
"editable": true
@@ -1010,7 +1010,7 @@
},
{
"cell_type": "markdown",
"id": "5627f5ae",
"id": "a56de206",
"metadata": {
"editable": true
},
@@ -1021,7 +1021,7 @@
{
"cell_type": "code",
"execution_count": 7,
"id": "a7c8662a",
"id": "0f2ff030",
"metadata": {
"collapsed": false,
"editable": true
@@ -1033,7 +1033,7 @@
},
{
"cell_type": "markdown",
"id": "d03ee2a7",
"id": "68abe92d",
"metadata": {
"editable": true
},
@@ -1059,7 +1059,7 @@
},
{
"cell_type": "markdown",
"id": "44ce5095",
"id": "3cea1c5a",
"metadata": {
"editable": true
},
@@ -1072,7 +1072,7 @@
{
"cell_type": "code",
"execution_count": 8,
"id": "24ff3dd3",
"id": "f42e1d87",
"metadata": {
"collapsed": false,
"editable": true
doc/LectureNotes/_build/html/chapter1.html
+100 -118
View File
@@ -992,24 +992,6 @@ infile = open(data_path(&quot;MassEval2016.dat&quot;),&#39;r&#39;)
</div>
</div>
</div>
<p>Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various <strong>matplotlib</strong> commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from pylab import plt, mpl
plt.style.use(&#39;seaborn&#39;)
mpl.rcParams[&#39;font.family&#39;] = &#39;serif&#39;
def MakePlot(x,y, styles, labels, axlabels):
plt.figure(figsize=(10,6))
for i in range(len(x)):
plt.plot(x[i], y[i], styles[i], label = labels[i])
plt.xlabel(axlabels[0])
plt.ylabel(axlabels[1])
plt.legend(loc=0)
</pre></div>
</div>
</div>
</div>
<p>Our next step is to read the data on experimental binding energies and
reorganize them as functions of the mass number <span class="math notranslate nohighlight">\(A\)</span>, the number of
protons <span class="math notranslate nohighlight">\(Z\)</span> and neutrons <span class="math notranslate nohighlight">\(N\)</span> using <strong>pandas</strong>. Before we do this it is
@@ -1224,11 +1206,11 @@ plt.show()
<section id="linear-regression-basic-elements">
<h2><span class="section-number">3.4. </span>Linear Regression, basic elements<a class="headerlink" href="#linear-regression-basic-elements" title="Link to this heading">#</a></h2>
<p><a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureAug27.mp4?vrtx=view-as-webpage">Video of Lecture</a>.</p>
<p>Fitting a continuous function with linear parameterization in terms of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>Fitting a continuous function with linear parameterization in terms of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
<ul class="simple">
<li><p>Method of choice for fitting a continuous function!</p></li>
<li><p>Gives an excellent introduction to central Machine Learning features with <strong>understandable pedagogical</strong> links to other methods like <strong>Neural Networks</strong>, <strong>Support Vector Machines</strong> etc</p></li>
<li><p>Analytical expression for the fitting parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span></p></li>
<li><p>Analytical expression for the fitting parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></p></li>
<li><p>Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more</p></li>
<li><p>Analytical relation with probabilistic interpretations</p></li>
<li><p>Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics</p></li>
@@ -1239,12 +1221,12 @@ plt.show()
<p>For more discussions of Ridge and Lasso regression, <a class="reference external" href="https://arxiv.org/abs/1509.09169">Wessel van Wieringen’s</a> article is highly recommended.
Similarly, <a class="reference external" href="https://arxiv.org/abs/1803.08823">Mehta et al’s article</a> is also recommended.</p>
<p>Regression modeling deals with the description of the sampling distribution of a given random variable <span class="math notranslate nohighlight">\(y\)</span> and how it varies as function of another variable or a set of such variables <span class="math notranslate nohighlight">\(\boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T\)</span>.
The first variable is called the <strong>dependent</strong>, the <strong>outcome</strong> or the <strong>response</strong> variable while the set of variables <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> is called the independent variable, or the predictor variable or the explanatory variable.</p>
<p>A regression model aims at finding a likelihood function <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span>, that is the conditional distribution for <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with a given <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>. The estimation of <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span> is made using a data set with</p>
The first variable is called the <strong>dependent</strong>, the <strong>outcome</strong> or the <strong>response</strong> or just the output variable while the set of variables <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> is called the independent variable, or the predictor variable or the explanatory variable, or just the input variable. We will hereafter call <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}x\)</span> for the output and input variables, respectively.</p>
<p>A regression model aims at finding a likelihood function <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span> (or just a function <span class="math notranslate nohighlight">\(f(\boldsymbol{x}\)</span>) , that is the conditional distribution for <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with a given <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>. The estimation of <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span> is made using a data set with</p>
<ul class="simple">
<li><p><span class="math notranslate nohighlight">\(n\)</span> cases <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span></p></li>
<li><p>Response (target, dependent or outcome) variable <span class="math notranslate nohighlight">\(y_i\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span></p></li>
<li><p><span class="math notranslate nohighlight">\(p\)</span> so-called explanatory (independent or predictor) variables <span class="math notranslate nohighlight">\(\boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span> and explanatory variables running from <span class="math notranslate nohighlight">\(0\)</span> to <span class="math notranslate nohighlight">\(p-1\)</span>. See below for more explicit examples.</p></li>
<li><p>Response/output (target, dependent or outcome) variable <span class="math notranslate nohighlight">\(y_i\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span></p></li>
<li><p><span class="math notranslate nohighlight">\(p\)</span> so-called explanatory/input (independent or predictor) variables <span class="math notranslate nohighlight">\(\boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span> and explanatory variables running from <span class="math notranslate nohighlight">\(0\)</span> to <span class="math notranslate nohighlight">\(p-1\)</span>. See below for more explicit examples.</p></li>
</ul>
<p>The goal of the regression analysis is to extract/exploit relationship between <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.</p>
<p>Consider an experiment in which <span class="math notranslate nohighlight">\(p\)</span> characteristics of <span class="math notranslate nohighlight">\(n\)</span> samples are
@@ -1259,9 +1241,9 @@ regression analysis is to explain <span class="math notranslate nohighlight">\(\
f(\mathbf{X}_{i,\ast})\)</span>. When no prior knowledge on the form of
<span class="math notranslate nohighlight">\(f(\cdot)\)</span> is available, it is common to assume a linear relationship
between <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. This assumption gives rise to
the <em>linear regression model</em> where <span class="math notranslate nohighlight">\(\boldsymbol{\beta} = [\beta_0, \ldots,
\beta_{p-1}]^{T}\)</span> are the <em>regression parameters</em>.</p>
<p>Linear regression gives us a set of analytical equations for the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span>.</p>
the <em>linear regression model</em> where <span class="math notranslate nohighlight">\(\boldsymbol{\theta} = [\theta_0, \ldots,
\theta_{p-1}]^{T}\)</span> are the <em>regression parameters</em>.</p>
<p>Linear regression gives us a set of analytical equations for the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span>.</p>
<p>In order to understand the relation among the predictors <span class="math notranslate nohighlight">\(p\)</span>, the set of data <span class="math notranslate nohighlight">\(n\)</span> and the target (outcome, output etc) <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>,
consider the model we discussed for describing nuclear binding energies.</p>
<p>There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model.
@@ -1279,18 +1261,18 @@ so-called <a class="reference external" href="https://www.sciencedirect.com/scie
<p>Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of <span class="math notranslate nohighlight">\(y\)</span> which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree <span class="math notranslate nohighlight">\(n-1\)</span> with <span class="math notranslate nohighlight">\(n\)</span> points, that is</p>
<div class="math notranslate nohighlight">
\[
y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i,
y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \theta_j x_i^j+\epsilon_i,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\epsilon_i\)</span> is the error in our approximation.</p>
<p>For every set of values <span class="math notranslate nohighlight">\(y_i,x_i\)</span> we have thus the corresponding set of equations</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
y_0&amp;=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&amp;=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&amp;=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
y_0&amp;=\theta_0+\theta_1x_0^1+\theta_2x_0^2+\dots+\theta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&amp;=\theta_0+\theta_1x_1^1+\theta_2x_1^2+\dots+\theta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&amp;=\theta_0+\theta_1x_2^1+\theta_2x_2^2+\dots+\theta_{n-1}x_2^{n-1}+\epsilon_2\\
\dots &amp; \dots \\
y_{n-1}&amp;=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\
y_{n-1}&amp;=\theta_0+\theta_1x_{n-1}^1+\theta_2x_{n-1}^2+\dots+\theta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\
\end{align*}
\end{split}\]</div>
<p>Defining the vectors</p>
@@ -1301,7 +1283,7 @@ y_{n-1}&amp;=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^
<p>and</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T,
\boldsymbol{\theta} = [\theta_0,\theta_1, \theta_2,\dots, \theta_{n-1}]^T,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
@@ -1323,7 +1305,7 @@ y_{n-1}&amp;=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^
<p>we can rewrite our equations as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}.
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta}+\boldsymbol{\epsilon}.
\]</div>
<p>The above design matrix is called a <a class="reference external" href="https://en.wikipedia.org/wiki/Vandermonde_matrix">Vandermonde matrix</a>.</p>
<p>We are obviously not limited to the above polynomial expansions. We
@@ -1334,13 +1316,13 @@ of values <span class="math notranslate nohighlight">\(y_i,x_i\)</span> we can t
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
y_0&amp;=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
y_0&amp;=\theta_0x_{00}+\theta_1x_{01}+\theta_2x_{02}+\dots+\theta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\theta_0x_{10}+\theta_1x_{11}+\theta_2x_{12}+\dots+\theta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\theta_0x_{20}+\theta_1x_{21}+\theta_2x_{22}+\dots+\theta_{n-1}x_{2n-1}+\epsilon_2\\
\dots &amp; \dots \\
y_{i}&amp;=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
y_{i}&amp;=\theta_0x_{i0}+\theta_1x_{i1}+\theta_2x_{i2}+\dots+\theta_{n-1}x_{in-1}+\epsilon_i\\
\dots &amp; \dots \\
y_{n-1}&amp;=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
y_{n-1}&amp;=\theta_0x_{n-1,0}+\theta_1x_{n-1,2}+\theta_2x_{n-1,2}+\dots+\theta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
\end{split}\]</div>
<p><strong>Note that we have <span class="math notranslate nohighlight">\(p=n\)</span> here. The matrix is symmetric. This is generally not the case!</strong></p>
@@ -1359,20 +1341,20 @@ x_{n-1,0}&amp; x_{n-1,1} &amp;x_{n-1,2}&amp; \dots &amp; \dots &amp;x_{n-1,n-1}\
<p>and without loss of generality we rewrite again our equations as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}.
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta}+\boldsymbol{\epsilon}.
\]</div>
<p>The left-hand side of this equation is kwown. Our error vector <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> and the parameter vector <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> are our unknow quantities. How can we obtain the optimal set of <span class="math notranslate nohighlight">\(\beta_i\)</span> values?</p>
<p>The left-hand side of this equation is kwown. Our error vector <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> and the parameter vector <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> are our unknow quantities. How can we obtain the optimal set of <span class="math notranslate nohighlight">\(\theta_i\)</span> values?</p>
<p>We have defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> via the equations</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
y_0&amp;=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
y_0&amp;=\theta_0x_{00}+\theta_1x_{01}+\theta_2x_{02}+\dots+\theta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\theta_0x_{10}+\theta_1x_{11}+\theta_2x_{12}+\dots+\theta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\theta_0x_{20}+\theta_1x_{21}+\theta_2x_{22}+\dots+\theta_{n-1}x_{2n-1}+\epsilon_1\\
\dots &amp; \dots \\
y_{i}&amp;=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
y_{i}&amp;=\theta_0x_{i0}+\theta_1x_{i1}+\theta_2x_{i2}+\dots+\theta_{n-1}x_{in-1}+\epsilon_1\\
\dots &amp; \dots \\
y_{n-1}&amp;=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
y_{n-1}&amp;=\theta_0x_{n-1,0}+\theta_1x_{n-1,2}+\theta_2x_{n-1,2}+\dots+\theta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
\end{split}\]</div>
<p>As we noted above, we stayed with a system with the design matrix
@@ -1455,45 +1437,45 @@ display(DesignMatrix)
</div>
</div>
</div>
<p>With <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1}\)</span>, it means that we will hereafter write our equations for the approximation as</p>
<p>With <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\in {\mathbb{R}}^{p\times 1}\)</span>, it means that we will hereafter write our equations for the approximation as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>throughout these lectures.</p>
<p>With the above we use the design matrix to define the approximation <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> via the unknown quantity <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> as</p>
<p>With the above we use the design matrix to define the approximation <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> via the unknown quantity <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>and in order to find the optimal parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values <span class="math notranslate nohighlight">\(y_i\)</span> (which represent hopefully the exact values) and the parameterized values <span class="math notranslate nohighlight">\(\tilde{y}_i\)</span>, namely</p>
<p>and in order to find the optimal parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values <span class="math notranslate nohighlight">\(y_i\)</span> (which represent hopefully the exact values) and the parameterized values <span class="math notranslate nohighlight">\(\tilde{y}_i\)</span>, namely</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
\]</div>
<p>or using the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> and in a more compact matrix-vector notation as</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
\]</div>
<p>This function is one possible way to define the so-called cost function.</p>
<p>It is also common to define
the function <span class="math notranslate nohighlight">\(C\)</span> as</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
\]</div>
<p>since when taking the first derivative with respect to the unknown parameters <span class="math notranslate nohighlight">\(\beta\)</span>, the factor of <span class="math notranslate nohighlight">\(2\)</span> cancels out.</p>
<p>since when taking the first derivative with respect to the unknown parameters <span class="math notranslate nohighlight">\(\theta\)</span>, the factor of <span class="math notranslate nohighlight">\(2\)</span> cancels out.</p>
<p>The function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
\]</div>
<p>can be linked to the variance of the quantity <span class="math notranslate nohighlight">\(y_i\)</span> if we interpret the latter as the mean value.
When linking (see the discussion below) with the maximum likelihood approach below, we will indeed interpret <span class="math notranslate nohighlight">\(y_i\)</span> as a mean value</p>
<div class="math notranslate nohighlight">
\[
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\langle y_i \rangle\)</span> is the mean value. Keep in mind also that
till now we have treated <span class="math notranslate nohighlight">\(y_i\)</span> as the exact value. Normally, the
@@ -1503,41 +1485,41 @@ approximation to the true value. It is then always accompanied by an
error estimate, often limited to a statistical error estimate given by
the standard deviation discussed earlier. In the discussion here we
will treat <span class="math notranslate nohighlight">\(y_i\)</span> as our exact value for the response variable.</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(C(\boldsymbol{\beta})\)</span>, that is we are going to solve the problem</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(C(\boldsymbol{\theta})\)</span>, that is we are going to solve the problem</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
{\displaystyle \min_{\boldsymbol{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
\]</div>
<p>In practical terms it means we will require</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
\]</div>
<p>which results in</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
\]</div>
<p>or in a matrix-vector form as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
\]</div>
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
\]</div>
<p>as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>and if the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> is invertible we have the solution</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>We note also that since our design matrix is defined as <span class="math notranslate nohighlight">\(\boldsymbol{X}\in
{\mathbb{R}}^{n\times p}\)</span>, the product <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X} \in
@@ -1572,7 +1554,7 @@ We can then compute the second derivative of the cost function, which in our cas
of the means squared error. This leads to</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T\partial \boldsymbol{\theta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\]</div>
<p>This quantity defines the so- called the Hessian matrix.</p>
<p>The Hessian matrix plays an important role and is defined for the mean squared error as</p>
@@ -1587,28 +1569,28 @@ terms of the singular values.</p>
<p>The residuals <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> are in turn given by</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>and with</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
\]</div>
<p>we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
\]</div>
<p>meaning that the solution for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.</p>
<p>meaning that the solution for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.</p>
<p>Let us now return to our nuclear binding energies and simply code the above equations.</p>
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. After having defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> we simply need to
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. After having defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> we simply need to
write</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># matrix inversion to find beta
beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># matrix inversion to find theta
theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
# and then make the prediction
ytilde = X @ beta
ytilde = X @ theta
</pre></div>
</div>
</div>
@@ -1695,74 +1677,74 @@ response variable.</p>
as</p>
<div class="math notranslate nohighlight">
\[
\chi^2(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
\chi^2(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
\]</div>
<p>where the matrix <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> is a diagonal matrix with <span class="math notranslate nohighlight">\(\sigma_i\)</span> as matrix elements.</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(\chi^2(\boldsymbol{\beta})\)</span> by requiring</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(\chi^2(\boldsymbol{\theta})\)</span> by requiring</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
\]</div>
<p>which results in</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
\]</div>
<p>or in a matrix-vector form as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right).
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\theta}\right).
\]</div>
<p>where we have defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A} =\boldsymbol{X}/\boldsymbol{\Sigma}\)</span> with matrix elements <span class="math notranslate nohighlight">\(a_{ij} = x_{ij}/\sigma_i\)</span> and the vector <span class="math notranslate nohighlight">\(\boldsymbol{b}\)</span> with elements <span class="math notranslate nohighlight">\(b_i = y_i/\sigma_i\)</span>.</p>
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right),
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\theta}\right),
\]</div>
<p>as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\beta},
\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\theta},
\]</div>
<p>and if the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}^T\boldsymbol{A}\)</span> is invertible we have the solution</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}.
\boldsymbol{\theta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}.
\]</div>
<p>If we then introduce the matrix</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{H} = \left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1},
\]</div>
<p>we have then the following expression for the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> (the matrix elements of <span class="math notranslate nohighlight">\(\boldsymbol{H}\)</span> are <span class="math notranslate nohighlight">\(h_{ij}\)</span>)</p>
<p>we have then the following expression for the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> (the matrix elements of <span class="math notranslate nohighlight">\(\boldsymbol{H}\)</span> are <span class="math notranslate nohighlight">\(h_{ij}\)</span>)</p>
<div class="math notranslate nohighlight">
\[
\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
\theta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
\]</div>
<p>We state without proof the expression for the uncertainty in the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as (we leave this as an exercise)</p>
<p>We state without proof the expression for the uncertainty in the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> as (we leave this as an exercise)</p>
<div class="math notranslate nohighlight">
\[
\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
\sigma^2(\theta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \theta_j}{\partial y_i}\right)^2,
\]</div>
<p>resulting in</p>
<div class="math notranslate nohighlight">
\[
\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
\sigma^2(\theta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
\]</div>
<p>The first step here is to approximate the function <span class="math notranslate nohighlight">\(y\)</span> with a first-order polynomial, that is we write</p>
<div class="math notranslate nohighlight">
\[
y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
y=y(x) \rightarrow y(x_i) \approx \theta_0+\theta_1 x_i.
\]</div>
<p>By computing the derivatives of <span class="math notranslate nohighlight">\(\chi^2\)</span> with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> show that these are given by</p>
<p>By computing the derivatives of <span class="math notranslate nohighlight">\(\chi^2\)</span> with respect to <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> show that these are given by</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\theta_0-\theta_1x_{i}}{\sigma_i^2}\right)\right]=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\theta_0-\theta_1x_{i}}{\sigma_i^2}\right)\right]=0.
\]</div>
<p>For a linear fit (a first-order polynomial) we don’t need to invert a matrix!!<br />
Defining</p>
@@ -1789,15 +1771,15 @@ Defining</p>
<p>we obtain</p>
<div class="math notranslate nohighlight">
\[
\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
\theta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
\]</div>
<div class="math notranslate nohighlight">
\[
\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
\theta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
\]</div>
<p>This approach (different linear and non-linear regression) suffers
often from both being underdetermined and overdetermined in the
unknown coefficients <span class="math notranslate nohighlight">\(\beta_i\)</span>. A better approach is to use the
unknown coefficients <span class="math notranslate nohighlight">\(\theta_i\)</span>. A better approach is to use the
Singular Value Decomposition (SVD) method discussed below. Or using
Lasso and Ridge regression. See below.</p>
</section>
@@ -1939,16 +1921,16 @@ X[:,1] = x
X[:,2] = x**2
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(beta)
# matrix inversion to find theta
theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(theta)
# and then make the prediction
ytilde = X_train @ beta
ytilde = X_train @ theta
print(&quot;Training R2&quot;)
print(R2(y_train,ytilde))
print(&quot;Training MSE&quot;)
print(MSE(y_train,ytilde))
ypredict = X_test @ beta
ypredict = X_test @ theta
print(&quot;Test R2&quot;)
print(R2(y_test,ypredict))
print(&quot;Test MSE&quot;)
@@ -2037,15 +2019,15 @@ X[:,3] = Density**(4.0/3.0)
X[:,4] = Density**(5.0/3.0)
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
# matrix inversion to find beta
beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
# matrix inversion to find theta
theta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
# and then make the prediction
ytilde = X_train @ beta
ytilde = X_train @ theta
print(&quot;Training R2&quot;)
print(R2(y_train,ytilde))
print(&quot;Training MSE&quot;)
print(MSE(y_train,ytilde))
ypredict = X_test @ beta
ypredict = X_test @ theta
print(&quot;Test R2&quot;)
print(R2(y_test,ypredict))
print(&quot;Test MSE&quot;)
@@ -2506,7 +2488,7 @@ R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.</p>
<!-- --- begin solution of exercise --- -->
<p><strong>Solution.</strong>
The code here is an example of where we define our own design matrix and fit parameters <span class="math notranslate nohighlight">\(\beta\)</span>.</p>
The code here is an example of where we define our own design matrix and fit parameters <span class="math notranslate nohighlight">\(\theta\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import os
@@ -2535,16 +2517,16 @@ X[:,1] = x
X[:,2] = x**2
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(beta)
# matrix inversion to find theta
theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(theta)
# and then make the prediction
ytilde = X_train @ beta
ytilde = X_train @ theta
print(&quot;Training R2&quot;)
print(R2(y_train,ytilde))
print(&quot;Training MSE&quot;)
print(MSE(y_train,ytilde))
ypredict = X_test @ beta
ypredict = X_test @ theta
print(&quot;Test R2&quot;)
print(R2(y_test,ypredict))
print(&quot;Test MSE&quot;)
@@ -2755,16 +2737,16 @@ X[:,2] = x*x
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLSbeta)
# matrix inversion to find theta
OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLStheta)
# and then make the prediction
ytildeOLS = X_train @ OLSbeta
ytildeOLS = X_train @ OLStheta
print(&quot;Training R2 for OLS&quot;)
print(R2(y_train,ytildeOLS))
print(&quot;Training MSE for OLS&quot;)
print(MSE(y_train,ytildeOLS))
ypredictOLS = X_test @ OLSbeta
ypredictOLS = X_test @ OLStheta
print(&quot;Test R2 for OLS&quot;)
print(R2(y_test,ypredictOLS))
print(&quot;Test MSE OLS&quot;)
@@ -2781,10 +2763,10 @@ MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 1, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
OwnRidgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
# and then make the prediction
OwnytildeRidge = X_train @ OwnRidgebeta
OwnypredictRidge = X_test @ OwnRidgebeta
OwnytildeRidge = X_train @ OwnRidgetheta
OwnypredictRidge = X_test @ OwnRidgetheta
OwnMSEPredict[i] = MSE(y_test,OwnypredictRidge)
OwnMSETrain[i] = MSE(y_train,OwnytildeRidge)
# Make the fit using Ridge from Sklearn
doc/LectureNotes/_build/html/chapter2.html
+188 -188
View File
@@ -405,16 +405,16 @@ doconce format html chapter2.do.txt --><section class="tex2jax_ignore mathjax_i
<section id="mathematical-interpretation-of-ordinary-least-squares">
<h2><span class="section-number">4.1. </span>Mathematical Interpretation of Ordinary Least Squares<a class="headerlink" href="#mathematical-interpretation-of-ordinary-least-squares" title="Link to this heading">#</a></h2>
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).</p>
<p>We have shown that in ordinary least squares (OLS) the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> are given by</p>
<p>We have shown that in ordinary least squares (OLS) the optimal parameters <span class="math notranslate nohighlight">\(\theta\)</span> are given by</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>The <strong>hat</strong> over <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> means we have the optimal parameters after minimization of the cost function.</p>
<p>The <strong>hat</strong> over <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> means we have the optimal parameters after minimization of the cost function.</p>
<p>This means that our best model is defined as</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>We now define a matrix</p>
<div class="math notranslate nohighlight">
@@ -424,7 +424,7 @@ doconce format html chapter2.do.txt --><section class="tex2jax_ignore mathjax_i
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
\]</div>
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> has the important property that <span class="math notranslate nohighlight">\(\boldsymbol{A}^2=\boldsymbol{A}\)</span>. This is the definition of a <a class="reference external" href="https://en.wikipedia.org/wiki/Projection_matrix">projection matrix</a>.
We can then interpret our optimal model <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> as being represented by an orthogonal projection of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> onto a space defined by the column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. In our case here the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.</p>
@@ -525,15 +525,15 @@ This is equivalent to saying that the matrix <span class="math notranslate nohig
<div class="math notranslate nohighlight">
\[
\begin{equation}
\boldsymbol{\beta} = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
\boldsymbol{\theta} = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
\label{_auto1} \tag{1}
\end{equation}
\]</div>
<p>has linearly dependent column vectors, we will not be able to compute the inverse
of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\beta_i\)</span>.
of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\theta_i\)</span>.
The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})\)</span> can be inverted.
This is more likely to happen when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is high-dimensional. In this case it is likely to encounter a situation where
the regression parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> cannot be estimated.</p>
the regression parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> cannot be estimated.</p>
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
<div class="math notranslate nohighlight">
\[
@@ -923,7 +923,7 @@ least squares is proportional to the second derivative of the cost
function, that is we have</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T\partial \boldsymbol{\theta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\]</div>
<p>This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize).</p>
<p>The Hessian matrix plays an important role and is defined in this course as</p>
@@ -1154,7 +1154,7 @@ def create_X(x, y, n ):
y = np.ravel(y)
N = len(x)
l = int((n+1)*(n+2)/2) # Number of elements in beta
l = int((n+1)*(n+2)/2) # Number of elements in theta
X = np.ones((N,l))
for i in range(1,n+1):
@@ -1318,13 +1318,13 @@ values and the column vectors of <span class="math notranslate nohighlight">\(\b
our optimization problem is</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
\]</div>
<p>or we can state it as</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
{\displaystyle \min_{\boldsymbol{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
\]</div>
<p>where we have used the definition of a norm-2 vector, that is</p>
<div class="math notranslate nohighlight">
@@ -1332,26 +1332,26 @@ our optimization problem is</p>
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
\]</div>
<p>By minimizing the above equation with respect to the parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we could then obtain an analytical expression for the
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. We can add a regularization parameter <span class="math notranslate nohighlight">\(\lambda\)</span> by
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we could then obtain an analytical expression for the
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. We can add a regularization parameter <span class="math notranslate nohighlight">\(\lambda\)</span> by
defining a new cost function to be optimized, that is</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
{\displaystyle \min_{\boldsymbol{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
\]</div>
<p>which leads to the Ridge regression minimization problem where we
require that <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t\)</span>, where <span class="math notranslate nohighlight">\(t\)</span> is
require that <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t\)</span>, where <span class="math notranslate nohighlight">\(t\)</span> is
a finite number larger than zero. By defining</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
\]</div>
<p>we have a new optimization equation</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
{\displaystyle \min_{\boldsymbol{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
\]</div>
<p>which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.</p>
<p>Here we have defined the norm-1 as</p>
@@ -1362,44 +1362,44 @@ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\bolds
<p>Using the matrix-vector expression for Ridge regression and dropping the parameter <span class="math notranslate nohighlight">\(1/n\)</span> in front of the standard means squared error equation, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
\]</div>
<p>and
taking the derivatives with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we obtain then
taking the derivatives with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we obtain then
a slightly modified matrix inversion problem which for finite values
of <span class="math notranslate nohighlight">\(\lambda\)</span> does not suffer from singularity problems. We obtain
the optimal parameters</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\]</div>
<p>with <span class="math notranslate nohighlight">\(\boldsymbol{I}\)</span> being a <span class="math notranslate nohighlight">\(p\times p\)</span> identity matrix with the constraint that</p>
<div class="math notranslate nohighlight">
\[
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
\]</div>
<p>with <span class="math notranslate nohighlight">\(t\)</span> a finite positive number.</p>
<p>When we compare this with the ordinary least squares result we have</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\]</div>
<p>which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>.</p>
<p>We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>, we may
even reduce the variance of the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.</p>
even reduce the variance of the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.</p>
<p>Using our insights about the SVD of the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
\]</div>
<p>For Ridge regression this becomes</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
\]</div>
<p>with the vectors <span class="math notranslate nohighlight">\(\boldsymbol{u}_j\)</span> being the columns of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> from the SVD of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. Note that the sums goes to <span class="math notranslate nohighlight">\(p-1\)</span> since.</p>
<p>Since <span class="math notranslate nohighlight">\(\lambda \geq 0\)</span>, it means that compared to OLS, we have</p>
@@ -1421,12 +1421,12 @@ eigenvalues ordered in a descending way, that is <span class="math notranslate n
<p>In this case the standard OLS results in</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
\]</div>
<p>that is the Ridge estimator scales the OLS estimator by the inverse of a factor <span class="math notranslate nohighlight">\(1+\lambda\)</span>, and
the Ridge estimator converges to zero when the hyperparameter goes to
@@ -1435,64 +1435,64 @@ infinity.</p>
<p>Using the matrix-vector expression for Lasso regression and dropping the parameter <span class="math notranslate nohighlight">\(1/n\)</span> in front of the standard mean squared error equation, we have the following <strong>cost</strong> function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
\]</div>
<p>Taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
<p>Taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 &amp; \beta &gt; 0 \\-1 &amp; \beta &lt; 0, \end{array}\right.
\frac{d \vert \theta\vert}{d \boldsymbol{\theta}}=\mathrm{sgn}(\boldsymbol{\theta})=\left\{\begin{array}{cc} 1 &amp; \theta &gt; 0 \\-1 &amp; \theta &lt; 0, \end{array}\right.
\end{split}\]</div>
<p>we have that the derivative of the cost function is</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
\]</div>
<p>and reordering we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a class="reference external" href="https://cvxopt.org/">CVXOPT</a>. We will discuss this later.</p>
<p>Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
diagonal. In this case we have an equal number of rows and columns <span class="math notranslate nohighlight">\(n=p\)</span>.</p>
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\)</span> and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\theta}\)</span> and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2,
\]</div>
<p>and minimizing we have that</p>
<div class="math notranslate nohighlight">
\[
\hat{\beta}_i^{\mathrm{OLS}} = y_i.
\hat{\theta}_i^{\mathrm{OLS}} = y_i.
\]</div>
<p>For Ridge regression our cost function is</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\theta_i^2,
\]</div>
<p>and minimizing we have that</p>
<div class="math notranslate nohighlight">
\[
\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}.
\hat{\theta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}.
\]</div>
<p>For Lasso regression our cost function is</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\theta_i\vert=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\theta_i^2},
\]</div>
<p>and minimizing we have that</p>
<div class="math notranslate nohighlight">
\[
-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
-2\sum_{i=0}^{p-1}(y_i-\theta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\theta_i)}{\vert\theta_i\vert}=0,
\]</div>
<p>which leads to</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &amp;\mathrm{if} &amp; y_i&gt; \frac{\lambda}{2}\\
\hat{\boldsymbol{\theta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &amp;\mathrm{if} &amp; y_i&gt; \frac{\lambda}{2}\\
y_i+\frac{\lambda}{2} &amp;\mathrm{if} &amp; y_i&lt; -\frac{\lambda}{2}\\
0 &amp;\mathrm{if} &amp; \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
\end{split}\]</div>
<p>Plotting these results (<a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf">figure in handwritten notes for week 36</a>) shows clearly that Lasso regression suppresses (sets to zero) values of <span class="math notranslate nohighlight">\(\beta_i\)</span> for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Ridge regression reduces on the other hand the values of <span class="math notranslate nohighlight">\(\beta_i\)</span> as function of <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>Plotting these results (<a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf">figure in handwritten notes for week 36</a>) shows clearly that Lasso regression suppresses (sets to zero) values of <span class="math notranslate nohighlight">\(\theta_i\)</span> for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Ridge regression reduces on the other hand the values of <span class="math notranslate nohighlight">\(\theta_i\)</span> as function of <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>As another example,
let us assume we have a data set with outputs/targets given by the vector</p>
<div class="math notranslate nohighlight">
@@ -1504,99 +1504,99 @@ let us assume we have a data set with outputs/targets given by the vector</p>
\[\begin{split}
\boldsymbol{X}=\begin{bmatrix}2 &amp; 0\\ 0 &amp; 1 \\ 0 &amp; 0\end{bmatrix},
\end{split}\]</div>
<p>meaning that we have two features and two unknown parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> to be determined either by ordinary least squares, Ridge or Lasso regression.</p>
<p>meaning that we have two features and two unknown parameters <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> to be determined either by ordinary least squares, Ridge or Lasso regression.</p>
<p>For ordinary least squares (OLS) we know that the optimal solution is</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>Inserting the above values we obtain that</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
\end{split}\]</div>
<p>The code which implements this simpler case is presented after the discussion of Ridge and Lasso.</p>
<p>For Ridge regression we have</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\hat{\boldsymbol{\theta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>Inserting the above values we obtain that</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\hat{\boldsymbol{\theta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\end{split}\]</div>
<p>There is normally a constraint on the value of <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\beta}\vert\vert_2\)</span> via the parameter <span class="math notranslate nohighlight">\(\lambda\)</span>.
Let us for simplicity assume that <span class="math notranslate nohighlight">\(\beta_0^2+\beta_1^2=1\)</span> as constraint. This will allow us to find an expression for the optimal values of <span class="math notranslate nohighlight">\(\beta\)</span> and <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>There is normally a constraint on the value of <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\theta}\vert\vert_2\)</span> via the parameter <span class="math notranslate nohighlight">\(\lambda\)</span>.
Let us for simplicity assume that <span class="math notranslate nohighlight">\(\theta_0^2+\theta_1^2=1\)</span> as constraint. This will allow us to find an expression for the optimal values of <span class="math notranslate nohighlight">\(\theta\)</span> and <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>To see this, let us write the cost function for Ridge regression.</p>
<p>We define the MSE without the <span class="math notranslate nohighlight">\(1/n\)</span> factor and have then, using that</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix},
\boldsymbol{X}\boldsymbol{\theta}=\begin{bmatrix} 2\theta_0 \\ \theta_1 \\0 \end{bmatrix},
\end{split}\]</div>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2),
C(\boldsymbol{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\theta_0^2+\theta_1^2),
\]</div>
<p>and taking the derivative with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span> we get</p>
<p>and taking the derivative with respect to <span class="math notranslate nohighlight">\(\theta_0\)</span> we get</p>
<div class="math notranslate nohighlight">
\[
\beta_0=\frac{8}{4+\lambda},
\theta_0=\frac{8}{4+\lambda},
\]</div>
<p>and for <span class="math notranslate nohighlight">\(\beta_1\)</span> we obtain</p>
<p>and for <span class="math notranslate nohighlight">\(\theta_1\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\beta_1=\frac{2}{1+\lambda},
\theta_1=\frac{2}{1+\lambda},
\]</div>
<p>Using the constraint for <span class="math notranslate nohighlight">\(\beta_0^2+\beta_1^2=1\)</span> we can constrain <span class="math notranslate nohighlight">\(\lambda\)</span> by solving</p>
<p>Using the constraint for <span class="math notranslate nohighlight">\(\theta_0^2+\theta_1^2=1\)</span> we can constrain <span class="math notranslate nohighlight">\(\lambda\)</span> by solving</p>
<div class="math notranslate nohighlight">
\[
\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1,
\]</div>
<p>which gives <span class="math notranslate nohighlight">\(\lambda=4.571\)</span> and <span class="math notranslate nohighlight">\(\beta_0=0.933\)</span> and <span class="math notranslate nohighlight">\(\beta_1=0.359\)</span>.</p>
<p>For Lasso we need now, keeping a constraint on <span class="math notranslate nohighlight">\(\vert\beta_0\vert+\vert\beta_1\vert=1\)</span>, to take the derivative of the absolute values of <span class="math notranslate nohighlight">\(\beta_0\)</span>
and <span class="math notranslate nohighlight">\(\beta_1\)</span>. This gives us the following derivatives of the cost function</p>
<p>which gives <span class="math notranslate nohighlight">\(\lambda=4.571\)</span> and <span class="math notranslate nohighlight">\(\theta_0=0.933\)</span> and <span class="math notranslate nohighlight">\(\theta_1=0.359\)</span>.</p>
<p>For Lasso we need now, keeping a constraint on <span class="math notranslate nohighlight">\(\vert\theta_0\vert+\vert\theta_1\vert=1\)</span>, to take the derivative of the absolute values of <span class="math notranslate nohighlight">\(\theta_0\)</span>
and <span class="math notranslate nohighlight">\(\theta_1\)</span>. This gives us the following derivatives of the cost function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
C(\boldsymbol{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\vert\theta_0\vert+\vert\theta_1\vert),
\]</div>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_0}=-4(4-2\theta_0)+\lambda\mathrm{sgn}(\theta_0)=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_1}=-2(2-\theta_1)+\lambda\mathrm{sgn}(\theta_1)=0.
\]</div>
<p>We have now four cases to solve besides the trivial cases <span class="math notranslate nohighlight">\(\beta_0\)</span> and/or <span class="math notranslate nohighlight">\(\beta_1\)</span> are zero, namely</p>
<p>We have now four cases to solve besides the trivial cases <span class="math notranslate nohighlight">\(\theta_0\)</span> and/or <span class="math notranslate nohighlight">\(\theta_1\)</span> are zero, namely</p>
<ol class="arabic simple">
<li><p><span class="math notranslate nohighlight">\(\beta_0 &gt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &gt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\beta_0 &gt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &lt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\beta_0 &lt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &gt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\beta_0 &lt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &lt; 0\)</span>.</p></li>
<li><p><span class="math notranslate nohighlight">\(\theta_0 &gt; 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 &gt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\theta_0 &gt; 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 &lt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\theta_0 &lt; 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 &gt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\theta_0 &lt; 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 &lt; 0\)</span>.</p></li>
</ol>
<p>If we consider the first case, we have then</p>
<div class="math notranslate nohighlight">
\[
-4(4-2\beta_0)+\lambda=0,
-4(4-2\theta_0)+\lambda=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
-2(2-\beta_1)+\lambda=0.
-2(2-\theta_1)+\lambda=0.
\]</div>
<p>which yields</p>
<div class="math notranslate nohighlight">
\[
\beta_0=\frac{16+\lambda}{8},
\theta_0=\frac{16+\lambda}{8},
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\beta_1=\frac{4+\lambda}{2}.
\theta_1=\frac{4+\lambda}{2}.
\]</div>
<p>Using the constraint on <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> we can then find the optimal value of <span class="math notranslate nohighlight">\(\lambda\)</span> for the different cases. We leave this as an exercise to you.</p>
<p>Using the constraint on <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> we can then find the optimal value of <span class="math notranslate nohighlight">\(\lambda\)</span> for the different cases. We leave this as an exercise to you.</p>
<p>Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of <span class="math notranslate nohighlight">\(\lambda\)</span>, meaning that we need to perform a search in order to find the optimal values.</p>
<p>First we study and compare the OLS and Ridge results. The next code compares all three methods.
We select values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\in [10^{-4},10^4]\)</span> and compute the predicted values for ordinary least squares and Ridge regression.</p>
@@ -1623,14 +1623,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]])
y = np.array( [4, 2, 3])
# matrix inversion to find beta
OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
print(OLSbeta)
# matrix inversion to find theta
OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y
print(OLStheta)
# and then make the prediction
ytildeOLS = X @ OLSbeta
ytildeOLS = X @ OLStheta
print(&quot;Training MSE for OLS&quot;)
print(MSE(y,ytildeOLS))
ypredictOLS = X @ OLSbeta
ypredictOLS = X @ OLStheta
# Repeat now for Ridge regression and various values of the regularization parameter
I = np.eye(2,2)
@@ -1640,10 +1640,10 @@ MSEPredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 4, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
# print(Ridgebeta)
Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
# print(Ridgetheta)
# and then make the prediction
ypredictRidge = X @ Ridgebeta
ypredictRidge = X @ Ridgetheta
MSEPredict[i] = MSE(y,ypredictRidge)
# print(MSEPredict[i])
# Now plot the results
@@ -1657,9 +1657,9 @@ plt.show()
</div>
</div>
</div>
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
<p>This happens also for Lasso regression, as seen from the next code
output. The difference is that Lasso shrinks the values of <span class="math notranslate nohighlight">\(\beta\)</span> to
output. The difference is that Lasso shrinks the values of <span class="math notranslate nohighlight">\(\theta\)</span> to
zero at a much earlier stage and the results flatten out. We see that
Lasso gives also an excellent fit for small values of <span class="math notranslate nohighlight">\(\lambda\)</span> and
shows the best performance of the three regression methods.</p>
@@ -1685,14 +1685,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]])
y = np.array( [4, 2, 3])
# matrix inversion to find beta
OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
print(OLSbeta)
# matrix inversion to find theta
OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y
print(OLStheta)
# and then make the prediction
ytildeOLS = X @ OLSbeta
ytildeOLS = X @ OLStheta
print(&quot;Training MSE for OLS&quot;)
print(MSE(y,ytildeOLS))
ypredictOLS = X @ OLSbeta
ypredictOLS = X @ OLStheta
# Repeat now for Ridge regression and various values of the regularization parameter
I = np.eye(2,2)
@@ -1703,10 +1703,10 @@ MSELassoPredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 4, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
print(Ridgebeta)
Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
print(Ridgetheta)
# and then make the prediction
ypredictRidge = X @ Ridgebeta
ypredictRidge = X @ Ridgetheta
MSERidgePredict[i] = MSE(y,ypredictRidge)
RegLasso = linear_model.Lasso(lmb)
RegLasso.fit(X,y)
@@ -1730,7 +1730,7 @@ three regression methods. Depending on the level of noise, we note
that for small values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span> all three
methods produce the same mean squared error. Again, Lasso shrinks the
parameter values to zero much earlier than Ridge regression and the
Lasso results flatten out much earlier since all <span class="math notranslate nohighlight">\(\beta_j=0\)</span> (check
Lasso results flatten out much earlier since all <span class="math notranslate nohighlight">\(\theta_j=0\)</span> (check
this by printing the values). This case is an example of where OLS
performs best. Lasso and Ridge reproduce the OLS results for a limited
set of <span class="math notranslate nohighlight">\(\lambda\)</span> values.</p>
@@ -1767,14 +1767,14 @@ X[:,2] = x*x
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLSbeta)
# matrix inversion to find theta
OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLStheta)
# and then make the prediction
ytildeOLS = X_train @ OLSbeta
ytildeOLS = X_train @ OLStheta
print(&quot;Training MSE for OLS&quot;)
print(MSE(y_train,ytildeOLS))
ypredictOLS = X_test @ OLSbeta
ypredictOLS = X_test @ OLStheta
print(&quot;Test MSE OLS&quot;)
print(MSE(y_test,ypredictOLS))
@@ -1789,13 +1789,13 @@ MSELassoTrain = np.zeros(nlambdas)
lambdas = np.logspace(-4, 4, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
Ridgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
# include lasso using Scikit-Learn
RegLasso = linear_model.Lasso(lmb)
RegLasso.fit(X_train,y_train)
# and then make the prediction
ytildeRidge = X_train @ Ridgebeta
ypredictRidge = X_test @ Ridgebeta
ytildeRidge = X_train @ Ridgetheta
ypredictRidge = X_test @ Ridgetheta
ytildeLasso = RegLasso.predict(X_train)
ypredictLasso = RegLasso.predict(X_test)
MSEPredict[i] = MSE(y_test,ypredictRidge)
@@ -1832,7 +1832,7 @@ move from a linear algebra analysis to a statistical analysis. In
particular, we will focus on what the regularization terms can result
in. We will amongst other things show that the regularization
parameter can reduce considerably the variance of the parameters
<span class="math notranslate nohighlight">\(\beta\)</span>.</p>
<span class="math notranslate nohighlight">\(\theta\)</span>.</p>
<p>The
advantage of doing linear regression is that we actually end up with
analytical expressions for several statistical quantities.<br />
@@ -1853,7 +1853,7 @@ independent, i.e.:</p>
<p>The randomness of <span class="math notranslate nohighlight">\(\varepsilon_i\)</span> implies that
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is also a random variable. In particular,
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is normally distributed, because <span class="math notranslate nohighlight">\(\varepsilon_i \sim
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\)</span> is a
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\theta}\)</span> is a
non-random scalar. To specify the parameters of the distribution of
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> we need to calculate its first two moments.</p>
<p>Recall that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is a matrix of dimensionality <span class="math notranslate nohighlight">\(n\times p\)</span>. The
@@ -1870,15 +1870,15 @@ which describe our data</p>
function <span class="math notranslate nohighlight">\(f\)</span> is approximated by <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> where we want to minimize <span class="math notranslate nohighlight">\((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\)</span>, our MSE, with</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
\]</div>
<p>We can calculate the expectation value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> for a given element <span class="math notranslate nohighlight">\(i\)</span></p>
<div class="math notranslate nohighlight">
\[
\begin{align*}
\mathbb{E}(y_i) &amp; =
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
\end{align*}
\]</div>
<p>while
@@ -1888,80 +1888,80 @@ its variance is</p>
\begin{align*} \mbox{Var}(y_i) &amp; = \mathbb{E} \{ [y_i
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
[\mathbb{E}(y_i)]^2 \\ &amp; = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &amp;
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
\ast} \, \beta)^2 \\ &amp; = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &amp;
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
\ast} \, \theta)^2 \\ &amp; = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
\\ &amp; = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
\end{align*}
\end{split}\]</div>
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\beta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we can evaluate the expectation value</p>
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\theta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we can evaluate the expectation value</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
\mathbb{E}(\boldsymbol{\theta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
\]</div>
<p>This means that the estimator of the regression parameters is unbiased.</p>
<p>We can also calculate the variance</p>
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is</p>
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{eqnarray*}
\mbox{Var}(\boldsymbol{\beta}) &amp; = &amp; \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
\mbox{Var}(\boldsymbol{\theta}) &amp; = &amp; \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
\\
&amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
&amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
\\
% &amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
% &amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
% \\
% &amp; = &amp; \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
% &amp; = &amp; \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
% \\
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
\\
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
% \\
% &amp; = &amp; (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
% &amp; = &amp; (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
% \\
% &amp; &amp; + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
% &amp; &amp; + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
\\
&amp; = &amp; \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
&amp; = &amp; \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
\end{eqnarray*}
\end{split}\]</div>
<p>where we have used that <span class="math notranslate nohighlight">\(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\theta}) = \sigma^2
\, (\mathbf{X}^{T} \mathbf{X})^{-1}\)</span>, one obtains an estimate of the
variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient:
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
construct a confidence interval for the estimates.</p>
<p>In a similar way, we can obtain analytical expressions for say the
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and their variance
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and their variance
when we employ Ridge regression, allowing us again to define a confidence interval.</p>
<p>It is rather straightforward to show that</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
\]</div>
<p>We see clearly that
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span>. We say then that the ridge estimator is biased.</p>
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span>. We say then that the ridge estimator is biased.</p>
<p>We can also compute the variance as</p>
<div class="math notranslate nohighlight">
\[
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
\]</div>
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> goes to zero.</p>
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> goes to zero.</p>
<p>With this, we can compute the difference</p>
<div class="math notranslate nohighlight">
\[
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
\]</div>
<p>The difference is non-negative definite since each component of the
matrix product is non-negative definite.
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
</section>
<section id="deriving-ols-from-a-probability-distribution">
<h2><span class="section-number">4.12. </span>Deriving OLS from a probability distribution<a class="headerlink" href="#deriving-ols-from-a-probability-distribution" title="Link to this heading">#</a></h2>
@@ -1971,26 +1971,26 @@ that our output is determined by a given continuous function
distribution with zero mean value and an undetermined variance
<span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
<p>We found above that the outputs <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> have a mean value given by
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
the design matrix are not stochastic variables, we can assume that the
probability distribution of our targets is also a normal distribution
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span>. This means that a
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span>. This means that a
single output <span class="math notranslate nohighlight">\(y_i\)</span> is given by the Gaussian distribution</p>
<div class="math notranslate nohighlight">
\[
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
\]</div>
<p>We assume now that the various <span class="math notranslate nohighlight">\(y_i\)</span> values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as</p>
<div class="math notranslate nohighlight">
\[
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
\]</div>
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
<p>Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
\]</div>
<p>We will write this in a more compact form reserving <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> for the domain of events, including the ouputs (targets) and the inputs. That is
in case we have a simple one-dimensional input and output case</p>
@@ -2002,18 +2002,18 @@ in case we have a simple one-dimensional input and output case</p>
We can now rewrite the above probability as</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
\]</div>
<p>It is a conditional probability (see below) and reads as the
likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
<p>In statistics, maximum likelihood estimation (MLE) is a method of
estimating the parameters of an assumed probability distribution,
given some observed data. This is achieved by maximizing a likelihood
function so that, under the assumed statistical model, the observed
data is the most probable.</p>
<p>We will assume here that our events are given by the above Gaussian
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> by
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\theta\)</span> by
maximizing the above PDF. However, computing the derivatives of a
product function is cumbersome and can easily lead to overflow and/or
underflowproblems, with potentials for loss of numerical precision.</p>
@@ -2027,22 +2027,22 @@ is equivalent to the maximization/minimization of the function itself.</p>
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
\]</div>
<p>which becomes</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
\]</div>
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\beta\)</span> we recognize our familiar OLS equation, namely</p>
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\theta\)</span> we recognize our familiar OLS equation, namely</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
\]</div>
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\beta\)</span></p>
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\theta\)</span></p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
\]</div>
<p>Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.</p>
<p>A central theorem in statistics is Bayes’ theorem. This theorem plays a similar role as the good old Pythagoras’ theorem in geometry.
@@ -2160,9 +2160,9 @@ perhaps not that intuitive. The question is whether we can develop a
more intuitive way of understanding what Ridge and Lasso express.</p>
<p>Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.</p>
<p>We will play around with a study of the values for the optimal
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> using OLS, Ridge and Lasso regression. For
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> using OLS, Ridge and Lasso regression. For
OLS, you will notice as function of the noise and polynomial degree,
that the parameters <span class="math notranslate nohighlight">\(\beta\)</span> will fluctuate from order to order in the
that the parameters <span class="math notranslate nohighlight">\(\theta\)</span> will fluctuate from order to order in the
polynomial fit and that for larger and larger polynomial degrees of
freedom, the parameters will tend to increase in value for OLS.</p>
<p>For Ridge and Lasso regression, the higher order parameters will
@@ -2198,10 +2198,10 @@ for polydegree in range(1, Maxpolydegree):
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLSbeta)
ypredictOLS = X_test @ OLSbeta
# matrix inversion to find theta
OLStheta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLStheta)
ypredictOLS = X_test @ OLStheta
print(&quot;Test MSE OLS&quot;)
print(MSE(y_test,ypredictOLS))
# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn
@@ -2238,21 +2238,21 @@ plt.show()
</div>
</div>
<p>How can we understand this?</p>
<p>Let us write out the values of the coefficients <span class="math notranslate nohighlight">\(\beta_i\)</span> as functions
<p>Let us write out the values of the coefficients <span class="math notranslate nohighlight">\(\theta_i\)</span> as functions
of the polynomial degree and noise. We will focus only on the Ridge
results and some few selected values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>If we don’t include any noise and run this code for different values
of the polynomial degree, we notice that the results for <span class="math notranslate nohighlight">\(\beta_i\)</span> do
of the polynomial degree, we notice that the results for <span class="math notranslate nohighlight">\(\theta_i\)</span> do
not show great changes from one order to the next. This is an
indication that for higher polynomial orders, our parameters become
less important.</p>
<p>If we however add noise, what happens is that the polynomial fit is
trying to adjust the fit to traverse in the best possible way all data
points. This can lead to large fluctuations in the parameters
<span class="math notranslate nohighlight">\(\beta_i\)</span> as functions of polynomial order. It will also be reflected
in a larger value of the variance of each parameter <span class="math notranslate nohighlight">\(\beta_i\)</span>. What
<span class="math notranslate nohighlight">\(\theta_i\)</span> as functions of polynomial order. It will also be reflected
in a larger value of the variance of each parameter <span class="math notranslate nohighlight">\(\theta_i\)</span>. What
Ridge regression (and Lasso as well) are doing then is to try to
quench the fluctuations in the parameters of <span class="math notranslate nohighlight">\(\beta_i\)</span> which have a
quench the fluctuations in the parameters of <span class="math notranslate nohighlight">\(\theta_i\)</span> which have a
large variance (normally for higher orders in the polynomial).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -2291,26 +2291,26 @@ for i in range(nlambdas):
# and then make the prediction
ypredictRidge = RegRidge.predict(X_test)
Coeffs = np.array(RegRidge.coef_)
BetaValues = pd.DataFrame(Coeffs)
BetaValues.columns = [&#39;beta&#39;]
display(BetaValues)
ThetaValues = pd.DataFrame(Coeffs)
ThetaValues.columns = [&#39;theta&#39;]
display(ThetaValues)
</pre></div>
</div>
</div>
</div>
<p>As an exercise, repeat these calculations with ordinary least squares
only with and without noise. Calculate thereafter the variance of the
parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as function of polynomial order and of the added
parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> as function of polynomial order and of the added
noise. Here we recommend to use <span class="math notranslate nohighlight">\(\sigma^2=1\)</span> as variance for the
added noise (which follows a normal distribution with mean value zero).
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> vary more as function
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> vary more as function
of model complexity? And what about their variance?</p>
</section>
<section id="linking-bayes-theorem-with-ridge-and-lasso-regression">
<h2><span class="section-number">4.14. </span>Linking Bayes’ Theorem with Ridge and Lasso Regression<a class="headerlink" href="#linking-bayes-theorem-with-ridge-and-lasso-regression" title="Link to this heading">#</a></h2>
<p>We have seen that Ridge regression suppresses those features which
have a small singular value. This corresponds to a feature which exhibits
a large variance in the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span>.
a large variance in the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span>.
Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use
Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regression.</p>
<p>For ordinary least squares we postulated that the maximum likelihood for the domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> (one-dimensional case)</p>
@@ -2321,74 +2321,74 @@ Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regres
<p>is given by</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
\]</div>
<p>In Bayes’ theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> given a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span>? That is, how can we define the posterior probability</p>
<p>In Bayes’ theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> given a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span>? That is, how can we define the posterior probability</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta}\vert\boldsymbol{D}).
p(\boldsymbol{\theta}\vert\boldsymbol{D}).
\]</div>
<p>Bayes’ theorem comes to our rescue here since (omitting the normalization constant)</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
p(\boldsymbol{\theta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\theta})p(\boldsymbol{\theta}).
\]</div>
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\beta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\beta})\)</span>!</p>
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\theta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\theta})\)</span>!</p>
<p>With the posterior probability defined by a likelihood which we have
already modeled and an unknown prior, we are now ready to make
additional models for the prior.</p>
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and
the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and
the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is
given by a Gaussian with mean value zero and variance <span class="math notranslate nohighlight">\(\tau^2\)</span>, that</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
p(\boldsymbol{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}.
\]</div>
<p>Our posterior probability becomes then (omitting the normalization factor which is just a constant)</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
p(\boldsymbol{\theta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}.
\]</div>
<p>We can now optimize this quantity with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. As we
<p>We can now optimize this quantity with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. As we
did for OLS, this is most conveniently done by taking the negative
logarithm of the posterior probability. Doing so and leaving out the
terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
terms that do not depend on <span class="math notranslate nohighlight">\(\theta\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\theta}\vert\vert_2^2,
\]</div>
<p>and replacing <span class="math notranslate nohighlight">\(1/2\tau^2\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_2^2,
\]</div>
<p>which is our Ridge cost function! Nice, isn’t it?</p>
<p>To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (<a class="reference external" href="https://en.wikipedia.org/wiki/Laplace_distribution">Laplace in this case</a>) with zero mean value, that is</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}.
p(\boldsymbol{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}.
\]</div>
<p>Our posterior probability becomes then (omitting the normalization factor which is just a constant)</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}.
p(\boldsymbol{\theta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}.
\]</div>
<p>Taking the negative
logarithm of the posterior probability and leaving out the
constants terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
constants terms that do not depend on <span class="math notranslate nohighlight">\(\theta\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\theta}\vert\vert_1,
\]</div>
<p>and replacing <span class="math notranslate nohighlight">\(1/\tau\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
\]</div>
<p>which is our Lasso cost function!</p>
<p>Plotting these prior functions shows us that we can use the parameter
<span class="math notranslate nohighlight">\(\lambda\)</span> to shrink or increase the role of a given parameter
<span class="math notranslate nohighlight">\(\beta_j\)</span>. The variance for the Laplace distribution is
<span class="math notranslate nohighlight">\(\theta_j\)</span>. The variance for the Laplace distribution is
<span class="math notranslate nohighlight">\(2\tau^2=1/\lambda\)</span> while for the Gaussian distribution it is
<span class="math notranslate nohighlight">\(\sigma^2=1/(2\lambda)\)</span>. Thus, increasing the variance means
decreasing <span class="math notranslate nohighlight">\(\lambda\)</span> and shrinking the variance means increasing
doc/LectureNotes/_build/html/chapter3.html
+129 -129
View File
@@ -479,7 +479,7 @@ independent, i.e.:</p>
<p>The randomness of <span class="math notranslate nohighlight">\(\varepsilon_i\)</span> implies that
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is also a random variable. In particular,
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is normally distributed, because <span class="math notranslate nohighlight">\(\varepsilon_i \sim
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\)</span> is a
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\theta}\)</span> is a
non-random scalar. To specify the parameters of the distribution of
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> we need to calculate its first two moments.</p>
<p>Recall that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is a matrix of dimensionality <span class="math notranslate nohighlight">\(n\times p\)</span>. The
@@ -496,15 +496,15 @@ which describe our data</p>
function <span class="math notranslate nohighlight">\(f\)</span> is approximated by <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> where we want to minimize <span class="math notranslate nohighlight">\((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\)</span>, our MSE, with</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
\]</div>
<p>We can calculate the expectation value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> for a given element <span class="math notranslate nohighlight">\(i\)</span></p>
<div class="math notranslate nohighlight">
\[
\begin{align*}
\mathbb{E}(y_i) &amp; =
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
\end{align*}
\]</div>
<p>while
@@ -514,81 +514,81 @@ its variance is</p>
\begin{align*} \mbox{Var}(y_i) &amp; = \mathbb{E} \{ [y_i
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
[\mathbb{E}(y_i)]^2 \\ &amp; = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &amp;
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
\ast} \, \beta)^2 \\ &amp; = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &amp;
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
\ast} \, \theta)^2 \\ &amp; = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
\\ &amp; = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
\end{align*}
\end{split}\]</div>
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\beta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we can evaluate the expectation value</p>
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\theta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we can evaluate the expectation value</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
\mathbb{E}(\boldsymbol{\theta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
\]</div>
<p>This means that the estimator of the regression parameters is unbiased.</p>
<p>We can also calculate the variance</p>
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is</p>
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{eqnarray*}
\mbox{Var}(\boldsymbol{\beta}) &amp; = &amp; \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
\mbox{Var}(\boldsymbol{\theta}) &amp; = &amp; \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
\\
&amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
&amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
\\
% &amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
% &amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
% \\
% &amp; = &amp; \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
% &amp; = &amp; \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
% \\
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
\\
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
% \\
% &amp; = &amp; (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
% &amp; = &amp; (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
% \\
% &amp; &amp; + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
% &amp; &amp; + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
\\
&amp; = &amp; \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
&amp; = &amp; \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
\end{eqnarray*}
\end{split}\]</div>
<p>where we have used that <span class="math notranslate nohighlight">\(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\theta}) = \sigma^2
\, (\mathbf{X}^{T} \mathbf{X})^{-1}\)</span>, one obtains an estimate of the
variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient:
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 \sqrt{
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 \sqrt{
[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} }\)</span>. This may be used to
construct a confidence interval for the estimates.</p>
<p>In a similar way, we can obtain analytical expressions for say the
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and their variance
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and their variance
when we employ Ridge regression, allowing us again to define a confidence interval.</p>
<p>It is rather straightforward to show that</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
\]</div>
<p>We see clearly that
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span>. We say then that the ridge estimator is biased.</p>
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span>. We say then that the ridge estimator is biased.</p>
<p>We can also compute the variance as</p>
<div class="math notranslate nohighlight">
\[
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
\]</div>
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> goes to zero.</p>
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> goes to zero.</p>
<p>With this, we can compute the difference</p>
<div class="math notranslate nohighlight">
\[
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
\]</div>
<p>The difference is non-negative definite since each component of the
matrix product is non-negative definite.
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
</section>
<section id="id1">
<h2><span class="section-number">5.3. </span>Resampling methods<a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
@@ -620,7 +620,7 @@ satisfied, the methods will fail. Yet, it should be said that if the data are
independent, identically distributed, and we only want to estimate the
variance of <span class="math notranslate nohighlight">\(\overline{X}\)</span> (which often is the case), then there is no
need for bootstrapping.</p>
<p>The Jackknife works by making many replicas of the estimator <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>.
<p>The Jackknife works by making many replicas of the estimator <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>.
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,X_n)\)</span>.
Let <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> denote the vector</p>
<div class="math notranslate nohighlight">
@@ -629,8 +629,8 @@ Let <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> denot
\]</div>
<p>which equals the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> with the exception that observation
number <span class="math notranslate nohighlight">\(i\)</span> is left out. Using this notation, define
<span class="math notranslate nohighlight">\(\widehat{\beta}_i\)</span> to be the estimator
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> computed using <span class="math notranslate nohighlight">\(\vec{X}_i\)</span>.</p>
<span class="math notranslate nohighlight">\(\widehat{\theta}_i\)</span> to be the estimator
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> computed using <span class="math notranslate nohighlight">\(\vec{X}_i\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from numpy import *
@@ -677,27 +677,27 @@ advantages:</p>
<li><p>It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.</p></li>
<li><p>It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).</p></li>
</ol>
<p>Since <span class="math notranslate nohighlight">\(\widehat{\beta} = \widehat{\beta}(\boldsymbol{X})\)</span> is a function of random variables,
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> itself must be a random variable. Thus it has
<p>Since <span class="math notranslate nohighlight">\(\widehat{\theta} = \widehat{\theta}(\boldsymbol{X})\)</span> is a function of random variables,
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> itself must be a random variable. Thus it has
a pdf, call this function <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. The aim of the bootstrap is to
estimate <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> by the relative frequency of
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>. You can think of this as using a histogram
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>. You can think of this as using a histogram
in the place of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. If the relative frequency closely
resembles <span class="math notranslate nohighlight">\(p(\vec{t})\)</span>, then using numerics, it is straight forward to
estimate all the interesting parameters of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> using point
estimators.</p>
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> has
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> has
more than one component, and the components are independent, we use the
same estimator on each component separately. If the probability
density function of <span class="math notranslate nohighlight">\(X_i\)</span>, <span class="math notranslate nohighlight">\(p(x)\)</span>, had been known, then it would have
been straight forward to do this by:</p>
<ol class="arabic simple">
<li><p>Drawing lots of numbers from <span class="math notranslate nohighlight">\(p(x)\)</span>, suppose we call one such set of numbers <span class="math notranslate nohighlight">\((X_1^*, X_2^*, \cdots, X_n^*)\)</span>.</p></li>
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>.</p></li>
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>.</p></li>
</ol>
<p>By repeated use of (1) and (2), many
estimates of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> could have been obtained. The
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>
estimates of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> could have been obtained. The
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>
(think of a histogram) as an estimate of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>.</p>
<p>But
unless there is enough information available about the process that
@@ -715,18 +715,18 @@ frequency of the observation <span class="math notranslate nohighlight">\(X_i\)<
<ol class="arabic simple">
<li><p>Draw with replacement <span class="math notranslate nohighlight">\(n\)</span> numbers for the observed variables <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,x_n)\)</span>.</p></li>
<li><p>Define a vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> containing the values which were drawn from <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>.</p></li>
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \beta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \theta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
<li><p>Repeat this process <span class="math notranslate nohighlight">\(k\)</span> times.</p></li>
</ol>
<p>When you are done, you can draw a histogram of the relative frequency
of <span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>. This is your estimate of the probability
of <span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>. This is your estimate of the probability
distribution <span class="math notranslate nohighlight">\(p(t)\)</span>. Using this probability distribution you can
estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>. Instead
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>. Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of <span class="math notranslate nohighlight">\(\widehat
\beta\)</span>, apply the estimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
<span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>.</p>
\theta\)</span>, apply the estimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
<span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>.</p>
<p>Before we proceed however, we need to remind ourselves about a central
theorem in statistics, namely the so-called <strong>central limit theorem</strong>.
This theorem plays a central role in understanding why the Bootstrap
@@ -911,12 +911,12 @@ tasks. Consider a dataset <span class="math notranslate nohighlight">\(\mathcal{
<p>where <span class="math notranslate nohighlight">\(\epsilon\)</span> is normally distributed with mean zero and standard deviation <span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
<p>In our derivation of the ordinary least squares method we defined then
an approximation to the function <span class="math notranslate nohighlight">\(f\)</span> in terms of the parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}\)</span>.</p>
<p>Thereafter we found the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> by optimizing the means squared error via the so-called cost function</p>
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta}\)</span>.</p>
<p>Thereafter we found the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> by optimizing the means squared error via the so-called cost function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
\]</div>
<p>We can rewrite this as</p>
<div class="math notranslate nohighlight">
@@ -1267,20 +1267,20 @@ cross-validation (LOOCV).</p>
<div class="math notranslate nohighlight">
\[
\begin{align*}
\boldsymbol{\beta}_{-i}(\lambda) &amp; = ( \boldsymbol{X}_{-i, \ast}^{T}
\boldsymbol{\theta}_{-i}(\lambda) &amp; = ( \boldsymbol{X}_{-i, \ast}^{T}
\boldsymbol{X}_{-i, \ast} + \lambda \boldsymbol{I}_{pp})^{-1}
\boldsymbol{X}_{-i, \ast}^{T} \boldsymbol{y}_{-i}
\end{align*}
\]</div>
<ul class="simple">
<li><p>Evaluate the prediction performance of these models on the test set by <span class="math notranslate nohighlight">\(\log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}\)</span>. Or, by the prediction error <span class="math notranslate nohighlight">\(|y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\beta}_{-i}(\lambda)|\)</span>, the relative error, the error squared or the R2 score function.</p></li>
<li><p>Evaluate the prediction performance of these models on the test set by <span class="math notranslate nohighlight">\(\log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\theta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}\)</span>. Or, by the prediction error <span class="math notranslate nohighlight">\(|y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\theta}_{-i}(\lambda)|\)</span>, the relative error, the error squared or the R2 score function.</p></li>
<li><p>Repeat the first three steps such that each sample plays the role of the test set once.</p></li>
<li><p>Average the prediction performances of the test sets at each grid point of the penalty bias/parameter. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as</p></li>
</ul>
<div class="math notranslate nohighlight">
\[
\begin{align*}
\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}.
\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\theta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}.
\end{align*}
\]</div>
<p>For the various values of <span class="math notranslate nohighlight">\(k\)</span></p>
@@ -1483,8 +1483,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero
when all predictors are zero (the columns in the design matrix), it
may be a bad idea to implement a model which penalizes the intercept.
Furthermore, in for example Ridge and Lasso regression, the default solutions
from the library <strong>Scikit-Learn</strong> (when not shrinking <span class="math notranslate nohighlight">\(\beta_0\)</span>) for the unknown parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, are derived under the assumption that both <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and
from the library <strong>Scikit-Learn</strong> (when not shrinking <span class="math notranslate nohighlight">\(\theta_0\)</span>) for the unknown parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>, are derived under the assumption that both <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and
<span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> are zero centered, that is we subtract the mean values.</p>
<p>If our predictors represent different scales, then it is important to
standardize the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> by subtracting the mean of each
@@ -1537,39 +1537,39 @@ simplicity, we will focus on ordinary regression, as done in the above example.
<p>The cost/loss function for regression is</p>
<div class="math notranslate nohighlight">
\[
C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,.
\]</div>
<p>Recall also that we use the squared value. This expression can lead to an
increased penalty for higher differences between predicted and
output/target values.</p>
<p>What we have done is to single out the <span class="math notranslate nohighlight">\(\beta_0\)</span> term in the
<p>What we have done is to single out the <span class="math notranslate nohighlight">\(\theta_0\)</span> term in the
definition of the mean squared error (MSE). The design matrix <span class="math notranslate nohighlight">\(X\)</span>
does in this case not contain any intercept column. When we take the
derivative with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span>, we want the derivative to obey</p>
derivative with respect to <span class="math notranslate nohighlight">\(\theta_0\)</span>, we want the derivative to obey</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C}{\partial \beta_j} = 0,
\frac{\partial C}{\partial \theta_j} = 0,
\]</div>
<p>for all <span class="math notranslate nohighlight">\(j\)</span>. For <span class="math notranslate nohighlight">\(\beta_0\)</span> we have</p>
<p>for all <span class="math notranslate nohighlight">\(j\)</span>. For <span class="math notranslate nohighlight">\(\theta_0\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).
\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right).
\]</div>
<p>Multiplying away the constant <span class="math notranslate nohighlight">\(2/n\)</span>, we obtain</p>
<div class="math notranslate nohighlight">
\[
\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j.
\]</div>
<p>Let us specialize first to the case where we have only two parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>.
Our result for <span class="math notranslate nohighlight">\(\beta_0\)</span> simplifies then to</p>
<p>Let us specialize first to the case where we have only two parameters <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>.
Our result for <span class="math notranslate nohighlight">\(\theta_0\)</span> simplifies then to</p>
<div class="math notranslate nohighlight">
\[
n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1.
\]</div>
<p>We obtain then</p>
<div class="math notranslate nohighlight">
\[
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
\]</div>
<p>If we define</p>
<div class="math notranslate nohighlight">
@@ -1584,17 +1584,17 @@ n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
<p>we have</p>
<div class="math notranslate nohighlight">
\[
\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
\theta_0 = \mu_y - \theta_1\mu_{\boldsymbol{x}_1}.
\]</div>
<p>In the general case with more parameters than <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>, we have</p>
<p>In the general case with more parameters than <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j.
\]</div>
<p>We can rewrite the latter equation as</p>
<div class="math notranslate nohighlight">
\[
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\theta_j,
\]</div>
<p>where we have defined</p>
<div class="math notranslate nohighlight">
@@ -1605,19 +1605,19 @@ n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
<p>Replacing <span class="math notranslate nohighlight">\(y_i\)</span> with <span class="math notranslate nohighlight">\(y_i - y_i - \overline{\boldsymbol{y}}\)</span> and centering also our design matrix results in a cost function (in vector-matrix disguise)</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).
C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}).
\]</div>
<p>If we minimize with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we have then</p>
<p>If we minimize with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we have then</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}}\)</span>
and <span class="math notranslate nohighlight">\(\tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}\)</span>.</p>
<p>For Ridge regression we need to add <span class="math notranslate nohighlight">\(\lambda \boldsymbol{\beta}^T\boldsymbol{\beta}\)</span> to the cost function and get then</p>
<p>For Ridge regression we need to add <span class="math notranslate nohighlight">\(\lambda \boldsymbol{\theta}^T\boldsymbol{\theta}\)</span> to the cost function and get then</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
\]</div>
<p>What does this mean? And why do we insist on all this? Let us look at some examples.</p>
<p>This code shows a simple first-order fit to a data set using the above transformed data, where we consider the role of the intercept first, by either excluding it or including it (<em>code example thanks to Øyvind Sigmundson Schøyen</em>). Here our scaling of the data is done by subtracting the mean values only.
@@ -1637,15 +1637,15 @@ def MSE(y_data,y_model):
return np.sum((y_data-y_model)**2)/n
def fit_beta(X, y):
def fit_theta(X, y):
return np.linalg.pinv(X.T @ X) @ X.T @ y
true_beta = [2, 0.5, 3.7]
true_theta = [2, 0.5, 3.7]
x = np.linspace(0, 1, 11)
y = np.sum(
np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0
np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0
) + 0.1 * np.random.normal(size=len(x))
degree = 3
@@ -1655,15 +1655,15 @@ X = np.zeros((len(x), degree))
for p in range(degree):
X[:, p] = x ** p
beta = fit_beta(X, y)
theta = fit_theta(X, y)
# Intercept is included in the design matrix
skl = LinearRegression(fit_intercept=False).fit(X, y)
print(f&quot;True beta: {true_beta}&quot;)
print(f&quot;Fitted beta: {beta}&quot;)
print(f&quot;Sklearn fitted beta: {skl.coef_}&quot;)
ypredictOwn = X @ beta
print(f&quot;True theta: {true_theta}&quot;)
print(f&quot;Fitted theta: {theta}&quot;)
print(f&quot;Sklearn fitted theta: {skl.coef_}&quot;)
ypredictOwn = X @ theta
ypredictSKL = skl.predict(X)
print(f&quot;MSE with intercept column&quot;)
print(MSE(y,ypredictOwn))
@@ -1673,7 +1673,7 @@ print(MSE(y,ypredictSKL))
plt.figure()
plt.scatter(x, y, label=&quot;Data&quot;)
plt.plot(x, X @ beta, label=&quot;Fit&quot;)
plt.plot(x, X @ theta, label=&quot;Fit&quot;)
plt.plot(x, skl.predict(X), label=&quot;Sklearn (fit_intercept=False)&quot;)
@@ -1690,21 +1690,21 @@ skl = LinearRegression(fit_intercept=True).fit(X, y)
y_offset = np.average(y, axis=0)
X_offset = np.average(X, axis=0)
beta = fit_beta(X - X_offset, y - y_offset)
intercept = np.mean(y_offset - X_offset @ beta)
theta = fit_theta(X - X_offset, y - y_offset)
intercept = np.mean(y_offset - X_offset @ theta)
print(f&quot;Manual intercept: {intercept}&quot;)
print(f&quot;Fitted beta (wiothout intercept): {beta}&quot;)
print(f&quot;Fitted theta (wiothout intercept): {theta}&quot;)
print(f&quot;Sklearn intercept: {skl.intercept_}&quot;)
print(f&quot;Sklearn fitted beta (without intercept): {skl.coef_}&quot;)
ypredictOwn = X @ beta
print(f&quot;Sklearn fitted theta (without intercept): {skl.coef_}&quot;)
ypredictOwn = X @ theta
ypredictSKL = skl.predict(X)
print(f&quot;MSE with Manual intercept&quot;)
print(MSE(y,ypredictOwn+intercept))
print(f&quot;MSE with Sklearn intercept&quot;)
print(MSE(y,ypredictSKL))
plt.plot(x, X @ beta + intercept, &quot;--&quot;, label=&quot;Fit (manual intercept)&quot;)
plt.plot(x, X @ theta + intercept, &quot;--&quot;, label=&quot;Fit (manual intercept)&quot;)
plt.plot(x, skl.predict(X), &quot;--&quot;, label=&quot;Sklearn (fit_intercept=True)&quot;)
plt.grid()
plt.legend()
@@ -1721,22 +1721,22 @@ they should. However, when we move to for example Ridge regression,
the way we treat the intercept may give a larger or smaller MSE,
meaning that the MSE can be penalized by the value of the
intercept. Not including the intercept in the fit, means that the
regularization term does not include <span class="math notranslate nohighlight">\(\beta_0\)</span>. For different values
regularization term does not include <span class="math notranslate nohighlight">\(\theta_0\)</span>. For different values
of <span class="math notranslate nohighlight">\(\lambda\)</span>, this may lead to different MSE values.</p>
<p>To remind the reader, the regularization term, with the intercept in Ridge regression, is given by</p>
<div class="math notranslate nohighlight">
\[
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,
\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2,
\]</div>
<p>but when we take out the intercept, this equation becomes</p>
<div class="math notranslate nohighlight">
\[
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.
\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2.
\]</div>
<p>For Lasso regression we have</p>
<div class="math notranslate nohighlight">
\[
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
\lambda \vert\vert \boldsymbol{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert.
\]</div>
<p>It means that, when scaling the design matrix and the outputs/targets,
by subtracting the mean values, we have an optimization problem which
@@ -1783,20 +1783,20 @@ MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
# Note: we include the intercept column and no scaling
RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
RegRidge.fit(X_train,y_train)
# and then make the prediction
ytildeOwnRidge = X_train @ OwnRidgeBeta
ypredictOwnRidge = X_test @ OwnRidgeBeta
ytildeOwnRidge = X_train @ OwnRidgeTheta
ypredictOwnRidge = X_test @ OwnRidgeTheta
ytildeRidge = RegRidge.predict(X_train)
ypredictRidge = RegRidge.predict(X_test)
MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
MSERidgePredict[i] = MSE(y_test,ypredictRidge)
print(&quot;Beta values for own Ridge implementation&quot;)
print(OwnRidgeBeta)
print(&quot;Beta values for Scikit-Learn Ridge implementation&quot;)
print(&quot;Theta values for own Ridge implementation&quot;)
print(OwnRidgeTheta)
print(&quot;Theta values for Scikit-Learn Ridge implementation&quot;)
print(RegRidge.coef_)
print(&quot;MSE values for own Ridge implementation&quot;)
print(MSEOwnRidgePredict[i])
@@ -1869,18 +1869,18 @@ MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data
OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data
#Add intercept to prediction
ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler
ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler
RegRidge = linear_model.Ridge(lmb)
RegRidge.fit(X_train,y_train)
ypredictRidge = RegRidge.predict(X_test)
MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
MSERidgePredict[i] = MSE(y_test,ypredictRidge)
print(&quot;Beta values for own Ridge implementation&quot;)
print(OwnRidgeBeta) #Intercept is given by mean of target variable
print(&quot;Beta values for Scikit-Learn Ridge implementation&quot;)
print(&quot;Theta values for own Ridge implementation&quot;)
print(OwnRidgeTheta) #Intercept is given by mean of target variable
print(&quot;Theta values for Scikit-Learn Ridge implementation&quot;)
print(RegRidge.coef_)
print(&#39;Intercept from own implementation:&#39;)
print(intercept_)
@@ -1907,7 +1907,7 @@ plt.show()
<p>We see here, when compared to the code which includes explicitely the
intercept column, that our MSE value is actually smaller. This is
because the regularization term does not include the intercept value
<span class="math notranslate nohighlight">\(\beta_0\)</span> in the fitting. This applies to Lasso regularization as
<span class="math notranslate nohighlight">\(\theta_0\)</span> in the fitting. This applies to Lasso regularization as
well. It means that our optimization is now done only with the
centered matrix and/or vector that enter the fitting procedure. Note
also that the problem with the intercept occurs mainly in these type
@@ -1994,7 +1994,7 @@ with the form utilized in linear regression, that is</p>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon},
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta} + \boldsymbol{\epsilon},
\label{_auto4} \tag{4}
\end{equation}
\]</div>
@@ -2016,15 +2016,15 @@ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
<div class="math notranslate nohighlight">
\[
\begin{equation}
C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}.
C(\boldsymbol{X}, \boldsymbol{\theta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})\right\}.
\label{_auto5} \tag{5}
\end{equation}
\]</div>
<p>We then find the extremal point of <span class="math notranslate nohighlight">\(C\)</span> by taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> as discussed above.
This yields the expression for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> to be</p>
<p>We then find the extremal point of <span class="math notranslate nohighlight">\(C\)</span> by taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> as discussed above.
This yields the expression for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> to be</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}},
\boldsymbol{\theta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}},
\]</div>
<p>which immediately imposes some requirements on <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> as there must exist
an inverse of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T \boldsymbol{X}\)</span>. If the expression we are modeling contains an
@@ -2047,10 +2047,10 @@ X_test_own = np.concatenate(
<p>Doing the inversion directly turns out to be a bad idea since the matrix
<span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> is singular. An alternative approach is to use the <strong>singular
value decomposition</strong>. Using the definition of the Moore-Penrose
pseudoinverse we can write the equation for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> as</p>
pseudoinverse we can write the equation for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y},
\boldsymbol{\theta} = \boldsymbol{X}^{+}\boldsymbol{y},
\]</div>
<p>where the pseudoinverse of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is given by</p>
<div class="math notranslate nohighlight">
@@ -2066,7 +2066,7 @@ where <span class="math notranslate nohighlight">\(X^{+} = V\Sigma^{+} U^T\)</sp
<div class="math notranslate nohighlight">
\[
\begin{equation}
\boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}.
\boldsymbol{\theta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}.
\label{_auto6} \tag{6}
\end{equation}
\]</div>
@@ -2087,7 +2087,7 @@ linear system as an equation would reduce this down to
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>beta = ols_svd(X_train_own,y_train)
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>theta = ols_svd(X_train_own,y_train)
</pre></div>
</div>
</div>
@@ -2095,7 +2095,7 @@ linear system as an equation would reduce this down to
<p>When extracting the <span class="math notranslate nohighlight">\(J\)</span>-matrix we need to make sure that we remove the intercept, as is done here</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>J = beta[1:].reshape(L, L)
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>J = theta[1:].reshape(L, L)
</pre></div>
</div>
</div>
@@ -2195,7 +2195,7 @@ with the form utilized in linear regression, viz.</p>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}.
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta} + \boldsymbol{\epsilon}.
\label{_auto10} \tag{10}
\end{equation}
\]</div>
@@ -2256,14 +2256,14 @@ plt.show()
<p>Having explored the ordinary least squares we move on to ridge
regression. In ridge regression we include a <strong>regularizer</strong>. This
involves a new cost function which leads to a new estimate for the
weights <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. This results in a penalized regression problem. The
weights <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. This results in a penalized regression problem. The
cost function is given by</p>
<!-- Equation labels as ordinary links -->
<div id="_auto11"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}
C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}.
C(\boldsymbol{X}, \boldsymbol{\theta}; \lambda) = (\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y}) + \lambda \boldsymbol{\theta}^T\boldsymbol{\theta}.
\label{_auto11} \tag{11}
\end{equation}
\]</div>
@@ -2291,7 +2291,7 @@ plt.show()
<div class="math notranslate nohighlight">
\[
\begin{equation}
C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\beta}^T\boldsymbol{\beta}}.
C(\boldsymbol{X}, \boldsymbol{\theta}; \lambda) = (\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\theta}^T\boldsymbol{\theta}}.
\label{_auto12} \tag{12}
\end{equation}
\]</div>
@@ -2505,7 +2505,7 @@ distribution <span class="math notranslate nohighlight">\(N(0,1)\)</span>.</p>
value decomposition from e.g., <strong>numpy</strong> ) or use your code from
homeworks 1 and 2 and perform a standard least square regression
analysis using polynomials in <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> up to fifth order. Find the
<a class="reference external" href="https://en.wikipedia.org/wiki/Confidence_interval">confidence intervals</a> of the parameters (estimators) <span class="math notranslate nohighlight">\(\beta\)</span> by computing their
<a class="reference external" href="https://en.wikipedia.org/wiki/Confidence_interval">confidence intervals</a> of the parameters (estimators) <span class="math notranslate nohighlight">\(\theta\)</span> by computing their
variances, evaluate the Mean Squared error (MSE)</p>
<div class="math notranslate nohighlight">
\[
@@ -2565,13 +2565,13 @@ dataset <span class="math notranslate nohighlight">\(\mathcal{L}\)</span> consis
deviation <span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
<p>In our derivation of the ordinary least squares method we defined then
an approximation to the function <span class="math notranslate nohighlight">\(f\)</span> in terms of the parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}\)</span>.</p>
<p>The parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> are in turn found by optimizing the means
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta}\)</span>.</p>
<p>The parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> are in turn found by optimizing the means
squared error via the so-called cost function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
\]</div>
<p>Here the expected value <span class="math notranslate nohighlight">\(\mathbb{E}\)</span> is the sample value.</p>
<p>Show that you can rewrite this as</p>
doc/LectureNotes/_build/html/chapter4.html
+32 -32
View File
@@ -403,7 +403,7 @@ independent variables <span class="math notranslate nohighlight">\(x_i\)</span>.
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
optimal parameters <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> to the mean squared error. If we can invert
optimal parameters <span class="math notranslate nohighlight">\(\hat{\theta}\)</span> to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.</p>
<p>Classification problems, however, are concerned with outcomes taking
@@ -423,7 +423,7 @@ failure etc.</p>
<p>Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters <span class="math notranslate nohighlight">\(\hat{\beta}\)</span>. The optimization of the
equation in the parameters <span class="math notranslate nohighlight">\(\hat{\theta}\)</span>. The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
@@ -462,12 +462,12 @@ weighted linear combination, namely</p>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon},
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\theta} + \boldsymbol{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> is a vector representing the possible outcomes, <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is our
<span class="math notranslate nohighlight">\(n\times p\)</span> design matrix and <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> represents our estimators/predictors.</p>
<span class="math notranslate nohighlight">\(n\times p\)</span> design matrix and <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> represents our estimators/predictors.</p>
<p>The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels <span class="math notranslate nohighlight">\(y_i\)</span> are discrete variables. A typical example is the credit
@@ -561,7 +561,7 @@ plt.show()
In standard linear regression with a linear dependence on <span class="math notranslate nohighlight">\(x\)</span>, we would write this in terms of our model</p>
<div class="math notranslate nohighlight">
\[
f(y_i\vert x_i)=\beta_0+\beta_1 x_i.
f(y_i\vert x_i)=\theta_0+\theta_1 x_i.
\]</div>
<p>This expression implies however that <span class="math notranslate nohighlight">\(f(y_i\vert x_i)\)</span> could take any
value from minus infinity to plus infinity. If we however let
@@ -656,19 +656,19 @@ plt.show()
</div>
</div>
</div>
<p>We assume now that we have two classes with <span class="math notranslate nohighlight">\(y_i\)</span> either <span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assume also that we have only two parameters <span class="math notranslate nohighlight">\(\beta\)</span> in our fitting of the Sigmoid function, that is we define probabilities</p>
<p>We assume now that we have two classes with <span class="math notranslate nohighlight">\(y_i\)</span> either <span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assume also that we have only two parameters <span class="math notranslate nohighlight">\(\theta\)</span> in our fitting of the Sigmoid function, that is we define probabilities</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
p(y_i=1|x_i,\boldsymbol{\beta}) &amp;= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\beta}) &amp;= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
p(y_i=1|x_i,\boldsymbol{\theta}) &amp;= \frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\theta}) &amp;= 1 - p(y_i=1|x_i,\boldsymbol{\theta}),
\end{align*}
\end{split}\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>.</p>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>.</p>
<p>Note that we used</p>
<div class="math notranslate nohighlight">
\[
p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
p(y_i=0\vert x_i, \boldsymbol{\theta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\theta}).
\]</div>
<p>In order to define the total likelihood for all possible outcomes from a<br />
dataset <span class="math notranslate nohighlight">\(\mathcal{D}=\{(y_i,x_i)\}\)</span>, with the binary labels
@@ -679,80 +679,80 @@ likelihood in terms of the product of the individual probabilities of a specific
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
P(\mathcal{D}|\boldsymbol{\beta})&amp; = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
P(\mathcal{D}|\boldsymbol{\theta})&amp; = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\theta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]^{1-y_i}\nonumber \\
\end{align*}
\end{split}\]</div>
<p>from which we obtain the log-likelihood and our <strong>cost/loss</strong> function</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\theta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]\right).
\]</div>
<p>Reordering the logarithms, we can rewrite the <strong>cost/loss</strong> function as</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
\]</div>
<p>The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to <span class="math notranslate nohighlight">\(\beta\)</span>.
<p>The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to <span class="math notranslate nohighlight">\(\theta\)</span>.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\theta})=-\sum_{i=1}^n \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
\]</div>
<p>This equation is known in statistics as the <strong>cross entropy</strong>. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually <span class="math notranslate nohighlight">\(L_1\)</span> and <span class="math notranslate nohighlight">\(L_2\)</span> regularization as we did for Ridge and Lasso regression.</p>
<p>The cross entropy is a convex function of the weights <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and,
<p>The cross entropy is a convex function of the weights <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and,
therefore, any local minimizer is a global minimizer.</p>
<p>Minimizing this
cost function with respect to the two parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> we obtain</p>
cost function with respect to the two parameters <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right),
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right).
\]</div>
<p>Let us now define a vector <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with <span class="math notranslate nohighlight">\(n\)</span> elements <span class="math notranslate nohighlight">\(y_i\)</span>, an
<span class="math notranslate nohighlight">\(n\times p\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which contains the <span class="math notranslate nohighlight">\(x_i\)</span> values and a
vector <span class="math notranslate nohighlight">\(\boldsymbol{p}\)</span> of fitted probabilities <span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\beta})\)</span>. We can rewrite in a more compact form the first
vector <span class="math notranslate nohighlight">\(\boldsymbol{p}\)</span> of fitted probabilities <span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})\)</span>. We can rewrite in a more compact form the first
derivative of cost function as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
\]</div>
<p>If we in addition define a diagonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{W}\)</span> with elements
<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta})\)</span>, we can obtain a compact expression of the second derivative as</p>
<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})(1-p(y_i\vert x_i,\boldsymbol{\theta})\)</span>, we can obtain a compact expression of the second derivative as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
\frac{\partial^2 \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
\]</div>
<p>Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with <span class="math notranslate nohighlight">\(p\)</span> predictors</p>
<div class="math notranslate nohighlight">
\[
\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\log{ \frac{p(\boldsymbol{\theta}\boldsymbol{x})}{1-p(\boldsymbol{\theta}\boldsymbol{x})}} = \theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p.
\]</div>
<p>Here we defined <span class="math notranslate nohighlight">\(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p]\)</span> leading to</p>
<p>Here we defined <span class="math notranslate nohighlight">\(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\theta}=[\theta_0, \theta_1, \dots, \theta_p]\)</span> leading to</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
p(\boldsymbol{\theta}\boldsymbol{x})=\frac{ \exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}{1+\exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}.
\]</div>
<p>Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to <span class="math notranslate nohighlight">\(K\)</span> classes. Let us for the sake
of simplicity assume we have only two predictors. We have then following model</p>
<div class="math notranslate nohighlight">
\[
\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \theta_{10}+\theta_{11}x_1,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \theta_{20}+\theta_{21}x_1,
\]</div>
<p>and so on till the class <span class="math notranslate nohighlight">\(C=K-1\)</span> class</p>
<div class="math notranslate nohighlight">
\[
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \theta_{(K-1)0}+\theta_{(K-1)1}x_1,
\]</div>
<p>and the model is specified in term of <span class="math notranslate nohighlight">\(K-1\)</span> so-called log-odds or
<strong>logit</strong> transformations.</p>
@@ -765,16 +765,16 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of <span class="math notranslate nohighlight">\(K\)</span> distinct linear functions,
and the predicted probability for the <span class="math notranslate nohighlight">\(k\)</span>-th class given a sample
vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> and a weighting vector <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is (with two
vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> and a weighting vector <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is (with two
predictors):</p>
<div class="math notranslate nohighlight">
\[
p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
p(C=k\vert \mathbf {x} )=\frac{\exp{(\theta_{k0}+\theta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}}.
\]</div>
<p>It is easy to extend to more predictors. The final class is</p>
<div class="math notranslate nohighlight">
\[
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}},
\]</div>
<p>and they sum to one. Our earlier discussions were all specialized to
the case with two classes only. It is easy to see from the above that
doc/LectureNotes/_build/html/genindex.html
+1 -1
View File
@@ -31,7 +31,7 @@
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
doc/LectureNotes/_build/html/intro.html
+1 -1
View File
@@ -32,7 +32,7 @@
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
doc/LectureNotes/_build/html/search.html
+1 -1
View File
@@ -30,7 +30,7 @@
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long