lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-08-18 10:47:43 +02:00
parent d4db3104e6
commit 35c45696d6
37 changed files with 5815 additions and 5344 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/E2.html
+20 -10
View File
@@ -469,37 +469,37 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
</section>
<section id="exercise-2-deriving-the-expression-for-ols">
<h2>Exercise 2 - Deriving the expression for OLS<a class="headerlink" href="#exercise-2-deriving-the-expression-for-ols" title="Link to this heading">#</a></h2>
<p>The ordinary least squares method finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes the squared error between our model <span class="math notranslate nohighlight">\(\boldsymbol{X\beta}\)</span> and the true values <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</p>
<p>To find the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes this error, we take the derivative of the squared error expression with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, and set it equal to 0.</p>
<p><strong>a)</strong> Very briefly explain why the approach above finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes this error.</p>
<p>The ordinary least squares method finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> which minimizes the squared error between our model <span class="math notranslate nohighlight">\(\boldsymbol{X\theta}\)</span> and the true values <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</p>
<p>To find the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> which minimizes this error, we take the derivative of the squared error expression with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>, and set it equal to 0.</p>
<p><strong>a)</strong> Very briefly explain why the approach above finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> which minimizes this error.</p>
<p>We typically write the squared error as</p>
<div class="math notranslate nohighlight">
\[
\vert\vert\boldsymbol{y} - \boldsymbol{X\beta}\vert\vert^2
\vert\vert\boldsymbol{y} - \boldsymbol{X\theta}\vert\vert^2
\]</div>
<p>which we can rewrite in matrix-vector form as</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)
\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)
\]</div>
<p><strong>b)</strong> If <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is invertible, what is the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>? (<strong>Hint:</strong> Don’t compute any derivatives, but solve <span class="math notranslate nohighlight">\(\boldsymbol{X\beta}=\boldsymbol{y}\)</span> for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>)</p>
<p><strong>b)</strong> If <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is invertible, what is the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>? (<strong>Hint:</strong> Don’t compute any derivatives, but solve <span class="math notranslate nohighlight">\(\boldsymbol{X\theta}=\boldsymbol{y}\)</span> for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>)</p>
<p><strong>c)</strong> Show that</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)}{\partial \boldsymbol{s}} = -2\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\boldsymbol{A},
\]</div>
<p><strong>d)</strong> Using the expression from <strong>c)</strong>, but substituting back in <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>, find the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> in the case that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is not invertible, but <span class="math notranslate nohighlight">\(\boldsymbol{X^T X}\)</span> is, which is most often the case.</p>
<p><strong>d)</strong> Using the expression from <strong>c)</strong>, but substituting back in <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>, <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>, find the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> in the case that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is not invertible, but <span class="math notranslate nohighlight">\(\boldsymbol{X^T X}\)</span> is, which is most often the case.</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\hat{\beta}_{OLS}} = ...
\boldsymbol{\hat{\theta}_{OLS}} = ...
\]</div>
</section>
<section id="exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression">
<h2>Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression<a class="headerlink" href="#exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression" title="Link to this heading">#</a></h2>
<p>With the expression for <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}_{OLS}}\)</span>, you now have what you need to implement OLS regression with your input data and target data <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. But before you can do that, you need to set up you input data as a feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>With the expression for <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\theta}_{OLS}}\)</span>, you now have what you need to implement OLS regression with your input data and target data <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. But before you can do that, you need to set up you input data as a feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.</p>
<p>We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.</p>
<p>We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter <span class="math notranslate nohighlight">\(\beta_0\)</span> from the OLS method into the optimal intercept value (which will be <span class="math notranslate nohighlight">\(\beta_0\)</span>). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.</p>
<p>We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter <span class="math notranslate nohighlight">\(\theta_0\)</span> from the OLS method into the optimal intercept value (which will be <span class="math notranslate nohighlight">\(\theta_0\)</span>). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
@@ -594,6 +594,11 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Ellipsis
</pre></div>
</div>
</div>
</div>
<p><strong>e)</strong> Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of <a class="reference external" href="https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf">Hastie et al</a>. Feel free to read the discussions leading to figure 2.11 of Hastie et al.</p>
<div class="cell docutils container">
@@ -602,6 +607,11 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Ellipsis
</pre></div>
</div>
</div>
</div>
<p><strong>f)</strong> Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?</p>
</section>