lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-08-18 10:47:43 +02:00
parent d4db3104e6
commit 35c45696d6
37 changed files with 5815 additions and 5344 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter1.html
+100 -118
View File
@@ -992,24 +992,6 @@ infile = open(data_path("MassEval2016.dat"),'r')
</div>
</div>
</div>
<p>Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various <strong>matplotlib</strong> commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from pylab import plt, mpl
plt.style.use(&#39;seaborn&#39;)
mpl.rcParams[&#39;font.family&#39;] = &#39;serif&#39;
def MakePlot(x,y, styles, labels, axlabels):
plt.figure(figsize=(10,6))
for i in range(len(x)):
plt.plot(x[i], y[i], styles[i], label = labels[i])
plt.xlabel(axlabels[0])
plt.ylabel(axlabels[1])
plt.legend(loc=0)
</pre></div>
</div>
</div>
</div>
<p>Our next step is to read the data on experimental binding energies and
reorganize them as functions of the mass number <span class="math notranslate nohighlight">\(A\)</span>, the number of
protons <span class="math notranslate nohighlight">\(Z\)</span> and neutrons <span class="math notranslate nohighlight">\(N\)</span> using <strong>pandas</strong>. Before we do this it is
@@ -1224,11 +1206,11 @@ plt.show()
<section id="linear-regression-basic-elements">
<h2><span class="section-number">3.4. </span>Linear Regression, basic elements<a class="headerlink" href="#linear-regression-basic-elements" title="Link to this heading">#</a></h2>
<p><a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureAug27.mp4?vrtx=view-as-webpage">Video of Lecture</a>.</p>
<p>Fitting a continuous function with linear parameterization in terms of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>Fitting a continuous function with linear parameterization in terms of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
<ul class="simple">
<li><p>Method of choice for fitting a continuous function!</p></li>
<li><p>Gives an excellent introduction to central Machine Learning features with <strong>understandable pedagogical</strong> links to other methods like <strong>Neural Networks</strong>, <strong>Support Vector Machines</strong> etc</p></li>
<li><p>Analytical expression for the fitting parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span></p></li>
<li><p>Analytical expression for the fitting parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></p></li>
<li><p>Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more</p></li>
<li><p>Analytical relation with probabilistic interpretations</p></li>
<li><p>Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics</p></li>
@@ -1239,12 +1221,12 @@ plt.show()
<p>For more discussions of Ridge and Lasso regression, <a class="reference external" href="https://arxiv.org/abs/1509.09169">Wessel van Wieringen’s</a> article is highly recommended.
Similarly, <a class="reference external" href="https://arxiv.org/abs/1803.08823">Mehta et al’s article</a> is also recommended.</p>
<p>Regression modeling deals with the description of the sampling distribution of a given random variable <span class="math notranslate nohighlight">\(y\)</span> and how it varies as function of another variable or a set of such variables <span class="math notranslate nohighlight">\(\boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T\)</span>.
The first variable is called the <strong>dependent</strong>, the <strong>outcome</strong> or the <strong>response</strong> variable while the set of variables <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> is called the independent variable, or the predictor variable or the explanatory variable.</p>
<p>A regression model aims at finding a likelihood function <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span>, that is the conditional distribution for <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with a given <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>. The estimation of <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span> is made using a data set with</p>
The first variable is called the <strong>dependent</strong>, the <strong>outcome</strong> or the <strong>response</strong> or just the output variable while the set of variables <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> is called the independent variable, or the predictor variable or the explanatory variable, or just the input variable. We will hereafter call <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}x\)</span> for the output and input variables, respectively.</p>
<p>A regression model aims at finding a likelihood function <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span> (or just a function <span class="math notranslate nohighlight">\(f(\boldsymbol{x}\)</span>) , that is the conditional distribution for <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with a given <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>. The estimation of <span class="math notranslate nohighlight">\(p(\boldsymbol{y}\vert \boldsymbol{x})\)</span> is made using a data set with</p>
<ul class="simple">
<li><p><span class="math notranslate nohighlight">\(n\)</span> cases <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span></p></li>
<li><p>Response (target, dependent or outcome) variable <span class="math notranslate nohighlight">\(y_i\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span></p></li>
<li><p><span class="math notranslate nohighlight">\(p\)</span> so-called explanatory (independent or predictor) variables <span class="math notranslate nohighlight">\(\boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span> and explanatory variables running from <span class="math notranslate nohighlight">\(0\)</span> to <span class="math notranslate nohighlight">\(p-1\)</span>. See below for more explicit examples.</p></li>
<li><p>Response/output (target, dependent or outcome) variable <span class="math notranslate nohighlight">\(y_i\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span></p></li>
<li><p><span class="math notranslate nohighlight">\(p\)</span> so-called explanatory/input (independent or predictor) variables <span class="math notranslate nohighlight">\(\boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]\)</span> with <span class="math notranslate nohighlight">\(i = 0, 1, 2, \dots, n-1\)</span> and explanatory variables running from <span class="math notranslate nohighlight">\(0\)</span> to <span class="math notranslate nohighlight">\(p-1\)</span>. See below for more explicit examples.</p></li>
</ul>
<p>The goal of the regression analysis is to extract/exploit relationship between <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.</p>
<p>Consider an experiment in which <span class="math notranslate nohighlight">\(p\)</span> characteristics of <span class="math notranslate nohighlight">\(n\)</span> samples are
@@ -1259,9 +1241,9 @@ regression analysis is to explain <span class="math notranslate nohighlight">\(\
f(\mathbf{X}_{i,\ast})\)</span>. When no prior knowledge on the form of
<span class="math notranslate nohighlight">\(f(\cdot)\)</span> is available, it is common to assume a linear relationship
between <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. This assumption gives rise to
the <em>linear regression model</em> where <span class="math notranslate nohighlight">\(\boldsymbol{\beta} = [\beta_0, \ldots,
\beta_{p-1}]^{T}\)</span> are the <em>regression parameters</em>.</p>
<p>Linear regression gives us a set of analytical equations for the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span>.</p>
the <em>linear regression model</em> where <span class="math notranslate nohighlight">\(\boldsymbol{\theta} = [\theta_0, \ldots,
\theta_{p-1}]^{T}\)</span> are the <em>regression parameters</em>.</p>
<p>Linear regression gives us a set of analytical equations for the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span>.</p>
<p>In order to understand the relation among the predictors <span class="math notranslate nohighlight">\(p\)</span>, the set of data <span class="math notranslate nohighlight">\(n\)</span> and the target (outcome, output etc) <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>,
consider the model we discussed for describing nuclear binding energies.</p>
<p>There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model.
@@ -1279,18 +1261,18 @@ so-called <a class="reference external" href="https://www.sciencedirect.com/scie
<p>Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of <span class="math notranslate nohighlight">\(y\)</span> which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree <span class="math notranslate nohighlight">\(n-1\)</span> with <span class="math notranslate nohighlight">\(n\)</span> points, that is</p>
<div class="math notranslate nohighlight">
\[
y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i,
y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \theta_j x_i^j+\epsilon_i,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\epsilon_i\)</span> is the error in our approximation.</p>
<p>For every set of values <span class="math notranslate nohighlight">\(y_i,x_i\)</span> we have thus the corresponding set of equations</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
y_0&amp;=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&amp;=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&amp;=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
y_0&amp;=\theta_0+\theta_1x_0^1+\theta_2x_0^2+\dots+\theta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&amp;=\theta_0+\theta_1x_1^1+\theta_2x_1^2+\dots+\theta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&amp;=\theta_0+\theta_1x_2^1+\theta_2x_2^2+\dots+\theta_{n-1}x_2^{n-1}+\epsilon_2\\
\dots &amp; \dots \\
y_{n-1}&amp;=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\
y_{n-1}&amp;=\theta_0+\theta_1x_{n-1}^1+\theta_2x_{n-1}^2+\dots+\theta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\
\end{align*}
\end{split}\]</div>
<p>Defining the vectors</p>
@@ -1301,7 +1283,7 @@ y_{n-1}&amp;=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^
<p>and</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T,
\boldsymbol{\theta} = [\theta_0,\theta_1, \theta_2,\dots, \theta_{n-1}]^T,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
@@ -1323,7 +1305,7 @@ y_{n-1}&amp;=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^
<p>we can rewrite our equations as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}.
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta}+\boldsymbol{\epsilon}.
\]</div>
<p>The above design matrix is called a <a class="reference external" href="https://en.wikipedia.org/wiki/Vandermonde_matrix">Vandermonde matrix</a>.</p>
<p>We are obviously not limited to the above polynomial expansions. We
@@ -1334,13 +1316,13 @@ of values <span class="math notranslate nohighlight">\(y_i,x_i\)</span> we can t
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
y_0&amp;=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
y_0&amp;=\theta_0x_{00}+\theta_1x_{01}+\theta_2x_{02}+\dots+\theta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\theta_0x_{10}+\theta_1x_{11}+\theta_2x_{12}+\dots+\theta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\theta_0x_{20}+\theta_1x_{21}+\theta_2x_{22}+\dots+\theta_{n-1}x_{2n-1}+\epsilon_2\\
\dots &amp; \dots \\
y_{i}&amp;=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
y_{i}&amp;=\theta_0x_{i0}+\theta_1x_{i1}+\theta_2x_{i2}+\dots+\theta_{n-1}x_{in-1}+\epsilon_i\\
\dots &amp; \dots \\
y_{n-1}&amp;=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
y_{n-1}&amp;=\theta_0x_{n-1,0}+\theta_1x_{n-1,2}+\theta_2x_{n-1,2}+\dots+\theta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
\end{split}\]</div>
<p><strong>Note that we have <span class="math notranslate nohighlight">\(p=n\)</span> here. The matrix is symmetric. This is generally not the case!</strong></p>
@@ -1359,20 +1341,20 @@ x_{n-1,0}&amp; x_{n-1,1} &amp;x_{n-1,2}&amp; \dots &amp; \dots &amp;x_{n-1,n-1}\
<p>and without loss of generality we rewrite again our equations as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}.
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta}+\boldsymbol{\epsilon}.
\]</div>
<p>The left-hand side of this equation is kwown. Our error vector <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> and the parameter vector <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> are our unknow quantities. How can we obtain the optimal set of <span class="math notranslate nohighlight">\(\beta_i\)</span> values?</p>
<p>The left-hand side of this equation is kwown. Our error vector <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> and the parameter vector <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> are our unknow quantities. How can we obtain the optimal set of <span class="math notranslate nohighlight">\(\theta_i\)</span> values?</p>
<p>We have defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> via the equations</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
y_0&amp;=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
y_0&amp;=\theta_0x_{00}+\theta_1x_{01}+\theta_2x_{02}+\dots+\theta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&amp;=\theta_0x_{10}+\theta_1x_{11}+\theta_2x_{12}+\dots+\theta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&amp;=\theta_0x_{20}+\theta_1x_{21}+\theta_2x_{22}+\dots+\theta_{n-1}x_{2n-1}+\epsilon_1\\
\dots &amp; \dots \\
y_{i}&amp;=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
y_{i}&amp;=\theta_0x_{i0}+\theta_1x_{i1}+\theta_2x_{i2}+\dots+\theta_{n-1}x_{in-1}+\epsilon_1\\
\dots &amp; \dots \\
y_{n-1}&amp;=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
y_{n-1}&amp;=\theta_0x_{n-1,0}+\theta_1x_{n-1,2}+\theta_2x_{n-1,2}+\dots+\theta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
\end{split}\]</div>
<p>As we noted above, we stayed with a system with the design matrix
@@ -1455,45 +1437,45 @@ display(DesignMatrix)
</div>
</div>
</div>
<p>With <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1}\)</span>, it means that we will hereafter write our equations for the approximation as</p>
<p>With <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\in {\mathbb{R}}^{p\times 1}\)</span>, it means that we will hereafter write our equations for the approximation as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>throughout these lectures.</p>
<p>With the above we use the design matrix to define the approximation <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> via the unknown quantity <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> as</p>
<p>With the above we use the design matrix to define the approximation <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> via the unknown quantity <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>and in order to find the optimal parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values <span class="math notranslate nohighlight">\(y_i\)</span> (which represent hopefully the exact values) and the parameterized values <span class="math notranslate nohighlight">\(\tilde{y}_i\)</span>, namely</p>
<p>and in order to find the optimal parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values <span class="math notranslate nohighlight">\(y_i\)</span> (which represent hopefully the exact values) and the parameterized values <span class="math notranslate nohighlight">\(\tilde{y}_i\)</span>, namely</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
\]</div>
<p>or using the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> and in a more compact matrix-vector notation as</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
\]</div>
<p>This function is one possible way to define the so-called cost function.</p>
<p>It is also common to define
the function <span class="math notranslate nohighlight">\(C\)</span> as</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
\]</div>
<p>since when taking the first derivative with respect to the unknown parameters <span class="math notranslate nohighlight">\(\beta\)</span>, the factor of <span class="math notranslate nohighlight">\(2\)</span> cancels out.</p>
<p>since when taking the first derivative with respect to the unknown parameters <span class="math notranslate nohighlight">\(\theta\)</span>, the factor of <span class="math notranslate nohighlight">\(2\)</span> cancels out.</p>
<p>The function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
\]</div>
<p>can be linked to the variance of the quantity <span class="math notranslate nohighlight">\(y_i\)</span> if we interpret the latter as the mean value.
When linking (see the discussion below) with the maximum likelihood approach below, we will indeed interpret <span class="math notranslate nohighlight">\(y_i\)</span> as a mean value</p>
<div class="math notranslate nohighlight">
\[
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\langle y_i \rangle\)</span> is the mean value. Keep in mind also that
till now we have treated <span class="math notranslate nohighlight">\(y_i\)</span> as the exact value. Normally, the
@@ -1503,41 +1485,41 @@ approximation to the true value. It is then always accompanied by an
error estimate, often limited to a statistical error estimate given by
the standard deviation discussed earlier. In the discussion here we
will treat <span class="math notranslate nohighlight">\(y_i\)</span> as our exact value for the response variable.</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(C(\boldsymbol{\beta})\)</span>, that is we are going to solve the problem</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(C(\boldsymbol{\theta})\)</span>, that is we are going to solve the problem</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
{\displaystyle \min_{\boldsymbol{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
\]</div>
<p>In practical terms it means we will require</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
\]</div>
<p>which results in</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
\]</div>
<p>or in a matrix-vector form as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
\]</div>
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
\]</div>
<p>as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>and if the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> is invertible we have the solution</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>We note also that since our design matrix is defined as <span class="math notranslate nohighlight">\(\boldsymbol{X}\in
{\mathbb{R}}^{n\times p}\)</span>, the product <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X} \in
@@ -1572,7 +1554,7 @@ We can then compute the second derivative of the cost function, which in our cas
of the means squared error. This leads to</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T\partial \boldsymbol{\theta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\]</div>
<p>This quantity defines the so- called the Hessian matrix.</p>
<p>The Hessian matrix plays an important role and is defined for the mean squared error as</p>
@@ -1587,28 +1569,28 @@ terms of the singular values.</p>
<p>The residuals <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> are in turn given by</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
\]</div>
<p>and with</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
\]</div>
<p>we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
\]</div>
<p>meaning that the solution for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.</p>
<p>meaning that the solution for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.</p>
<p>Let us now return to our nuclear binding energies and simply code the above equations.</p>
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. After having defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> we simply need to
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. After having defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> we simply need to
write</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># matrix inversion to find beta
beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># matrix inversion to find theta
theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
# and then make the prediction
ytilde = X @ beta
ytilde = X @ theta
</pre></div>
</div>
</div>
@@ -1695,74 +1677,74 @@ response variable.</p>
as</p>
<div class="math notranslate nohighlight">
\[
\chi^2(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
\chi^2(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
\]</div>
<p>where the matrix <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> is a diagonal matrix with <span class="math notranslate nohighlight">\(\sigma_i\)</span> as matrix elements.</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(\chi^2(\boldsymbol{\beta})\)</span> by requiring</p>
<p>In order to find the parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> we will then minimize the spread of <span class="math notranslate nohighlight">\(\chi^2(\boldsymbol{\theta})\)</span> by requiring</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
\]</div>
<p>which results in</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
\]</div>
<p>or in a matrix-vector form as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right).
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\theta}\right).
\]</div>
<p>where we have defined the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A} =\boldsymbol{X}/\boldsymbol{\Sigma}\)</span> with matrix elements <span class="math notranslate nohighlight">\(a_{ij} = x_{ij}/\sigma_i\)</span> and the vector <span class="math notranslate nohighlight">\(\boldsymbol{b}\)</span> with elements <span class="math notranslate nohighlight">\(b_i = y_i/\sigma_i\)</span>.</p>
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right),
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\theta}\right),
\]</div>
<p>as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\beta},
\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\theta},
\]</div>
<p>and if the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}^T\boldsymbol{A}\)</span> is invertible we have the solution</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}.
\boldsymbol{\theta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}.
\]</div>
<p>If we then introduce the matrix</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{H} = \left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1},
\]</div>
<p>we have then the following expression for the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> (the matrix elements of <span class="math notranslate nohighlight">\(\boldsymbol{H}\)</span> are <span class="math notranslate nohighlight">\(h_{ij}\)</span>)</p>
<p>we have then the following expression for the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> (the matrix elements of <span class="math notranslate nohighlight">\(\boldsymbol{H}\)</span> are <span class="math notranslate nohighlight">\(h_{ij}\)</span>)</p>
<div class="math notranslate nohighlight">
\[
\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
\theta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
\]</div>
<p>We state without proof the expression for the uncertainty in the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as (we leave this as an exercise)</p>
<p>We state without proof the expression for the uncertainty in the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> as (we leave this as an exercise)</p>
<div class="math notranslate nohighlight">
\[
\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
\sigma^2(\theta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \theta_j}{\partial y_i}\right)^2,
\]</div>
<p>resulting in</p>
<div class="math notranslate nohighlight">
\[
\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
\sigma^2(\theta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
\]</div>
<p>The first step here is to approximate the function <span class="math notranslate nohighlight">\(y\)</span> with a first-order polynomial, that is we write</p>
<div class="math notranslate nohighlight">
\[
y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
y=y(x) \rightarrow y(x_i) \approx \theta_0+\theta_1 x_i.
\]</div>
<p>By computing the derivatives of <span class="math notranslate nohighlight">\(\chi^2\)</span> with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> show that these are given by</p>
<p>By computing the derivatives of <span class="math notranslate nohighlight">\(\chi^2\)</span> with respect to <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> show that these are given by</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\theta_0-\theta_1x_{i}}{\sigma_i^2}\right)\right]=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
\frac{\partial \chi^2(\boldsymbol{\theta})}{\partial \theta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\theta_0-\theta_1x_{i}}{\sigma_i^2}\right)\right]=0.
\]</div>
<p>For a linear fit (a first-order polynomial) we don’t need to invert a matrix!!<br />
Defining</p>
@@ -1789,15 +1771,15 @@ Defining</p>
<p>we obtain</p>
<div class="math notranslate nohighlight">
\[
\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
\theta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
\]</div>
<div class="math notranslate nohighlight">
\[
\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
\theta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
\]</div>
<p>This approach (different linear and non-linear regression) suffers
often from both being underdetermined and overdetermined in the
unknown coefficients <span class="math notranslate nohighlight">\(\beta_i\)</span>. A better approach is to use the
unknown coefficients <span class="math notranslate nohighlight">\(\theta_i\)</span>. A better approach is to use the
Singular Value Decomposition (SVD) method discussed below. Or using
Lasso and Ridge regression. See below.</p>
</section>
@@ -1939,16 +1921,16 @@ X[:,1] = x
X[:,2] = x**2
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(beta)
# matrix inversion to find theta
theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(theta)
# and then make the prediction
ytilde = X_train @ beta
ytilde = X_train @ theta
print(&quot;Training R2&quot;)
print(R2(y_train,ytilde))
print(&quot;Training MSE&quot;)
print(MSE(y_train,ytilde))
ypredict = X_test @ beta
ypredict = X_test @ theta
print(&quot;Test R2&quot;)
print(R2(y_test,ypredict))
print(&quot;Test MSE&quot;)
@@ -2037,15 +2019,15 @@ X[:,3] = Density**(4.0/3.0)
X[:,4] = Density**(5.0/3.0)
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
# matrix inversion to find beta
beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
# matrix inversion to find theta
theta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
# and then make the prediction
ytilde = X_train @ beta
ytilde = X_train @ theta
print(&quot;Training R2&quot;)
print(R2(y_train,ytilde))
print(&quot;Training MSE&quot;)
print(MSE(y_train,ytilde))
ypredict = X_test @ beta
ypredict = X_test @ theta
print(&quot;Test R2&quot;)
print(R2(y_test,ypredict))
print(&quot;Test MSE&quot;)
@@ -2506,7 +2488,7 @@ R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.</p>
<!-- --- begin solution of exercise --- -->
<p><strong>Solution.</strong>
The code here is an example of where we define our own design matrix and fit parameters <span class="math notranslate nohighlight">\(\beta\)</span>.</p>
The code here is an example of where we define our own design matrix and fit parameters <span class="math notranslate nohighlight">\(\theta\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import os
@@ -2535,16 +2517,16 @@ X[:,1] = x
X[:,2] = x**2
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(beta)
# matrix inversion to find theta
theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(theta)
# and then make the prediction
ytilde = X_train @ beta
ytilde = X_train @ theta
print(&quot;Training R2&quot;)
print(R2(y_train,ytilde))
print(&quot;Training MSE&quot;)
print(MSE(y_train,ytilde))
ypredict = X_test @ beta
ypredict = X_test @ theta
print(&quot;Test R2&quot;)
print(R2(y_test,ypredict))
print(&quot;Test MSE&quot;)
@@ -2755,16 +2737,16 @@ X[:,2] = x*x
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLSbeta)
# matrix inversion to find theta
OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(OLStheta)
# and then make the prediction
ytildeOLS = X_train @ OLSbeta
ytildeOLS = X_train @ OLStheta
print(&quot;Training R2 for OLS&quot;)
print(R2(y_train,ytildeOLS))
print(&quot;Training MSE for OLS&quot;)
print(MSE(y_train,ytildeOLS))
ypredictOLS = X_test @ OLSbeta
ypredictOLS = X_test @ OLStheta
print(&quot;Test R2 for OLS&quot;)
print(R2(y_test,ypredictOLS))
print(&quot;Test MSE OLS&quot;)
@@ -2781,10 +2763,10 @@ MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 1, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
OwnRidgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
# and then make the prediction
OwnytildeRidge = X_train @ OwnRidgebeta
OwnypredictRidge = X_test @ OwnRidgebeta
OwnytildeRidge = X_train @ OwnRidgetheta
OwnypredictRidge = X_test @ OwnRidgetheta
OwnMSEPredict[i] = MSE(y_test,OwnypredictRidge)
OwnMSETrain[i] = MSE(y_train,OwnytildeRidge)
# Make the fit using Ridge from Sklearn