lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-01 08:51:55 +02:00
parent 7bcccbbf71
commit 7bd40a8c65
9 changed files with 589 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/exercisesweek37.html
+28 -28
View File
@@ -383,12 +383,12 @@ document.write(`
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#a">1a)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">2a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">2b)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#implementing-the-simplest-form-for-gradient-descent">Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-implementing-the-simplest-form-for-gradient-descent">Exercise 3, Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">3b)</a></li>
</ul>
@@ -459,13 +459,14 @@ y = X.dot @ theta_true + noise
</div>
<p>This code produces a dataset where only features 0, 1, and 6
significantly influence <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. The rest of the features have zero true
coefficient, so they only contribute noise. For example, feature 0 has
coefficient. For example, feature 0 has
a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so
the expected relationship is:</p>
<div class="math notranslate nohighlight">
\[
y \approx 5 \times X_0 \;-\; 3 \times X_1 \;+\; 2 \times X_6 \;+\; \text{noise}.
y \approx 5 \times x_0 \;-\; 3 \times x_1 \;+\; 2 \times x_6 \;+\; \text{noise}.
\]</div>
<p>You can remove the noise if you wish to.</p>
</section>
<section id="exercise-1-scale-your-data">
<h2>Exercise 1, scale your data<a class="headerlink" href="#exercise-1-scale-your-data" title="Link to this heading">#</a></h2>
@@ -473,12 +474,14 @@ y \approx 5 \times X_0 \;-\; 3 \times X_1 \;+\; 2 \times X_6 \;+\; \text{noise}.
standardize the features. This ensures all features are on a
comparable scale, which is especially important when using
regularization. Here we will perform standardization, scaling each
feature to have mean 0 and standard deviation 1:</p>
<p>Compute the mean and standard deviation of each column (feature) in <span class="math notranslate nohighlight">\(bm{X}\)</span>.
feature to have mean 0 and standard deviation 1.</p>
<section id="a">
<h3>1a)<a class="headerlink" href="#a" title="Link to this heading">#</a></h3>
<p>Compute the mean and standard deviation of each column (feature) in <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.
Subtract the mean and divide by the standard deviation for each feature.</p>
<p>We will also center the target <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> to mean <span class="math notranslate nohighlight">\(0\)</span>. Centering <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>
(and each feature) means the model won’t require a separate intercept
term – the data is shifted such that the intercept is effectively 0
(and each feature) means the model does not require a separate intercept
term, the data is shifted such that the intercept is effectively 0
. (In practice, one could include an intercept in the model and not
penalize it, but here we simplify by centering.)</p>
<div class="cell docutils container">
@@ -496,18 +499,16 @@ y_centered = ?
</div>
</div>
</div>
<section id="a">
<h3>1a)<a class="headerlink" href="#a" title="Link to this heading">#</a></h3>
<p>Fill in the necessary details.</p>
<p>After this preprocessing, each column of <span class="math notranslate nohighlight">\(\boldsymbol{X}_norm\)</span> has mean zero and standard deviation <span class="math notranslate nohighlight">\(1\)</span>
and <span class="math notranslate nohighlight">\(\boldsymbol{y}_centered\)</span> has mean 0. This makes the optimization landscape
<p>After this preprocessing, each column of <span class="math notranslate nohighlight">\(\boldsymbol{X}_{\mathrm{norm}}\)</span> has mean zero and standard deviation <span class="math notranslate nohighlight">\(1\)</span>
and <span class="math notranslate nohighlight">\(\boldsymbol{y}_{\mathrm{centered}}\)</span> has mean 0. This makes the optimization landscape
nicer and ensures the regularization penalty <span class="math notranslate nohighlight">\(\lambda \sum_j
\beta_j^2\)</span> treats each coefficient fairly (since features are on the
\theta_j^2\)</span> in Ridge regression treats each coefficient fairly (since features are on the
same scale).</p>
</section>
</section>
<section id="exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{theta}\)</span><a class="headerlink" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<h2>Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Set regularization parameter, either a single value or a vector of values
@@ -524,23 +525,22 @@ print(&quot;Closed-form OLS coefficients:&quot;, theta_closed_form)
</div>
</div>
</div>
<p>This computes the ridge and OLS regression coefficients directly. The identity
matrix <span class="math notranslate nohighlight">\(I\)</span> has the same size as <span class="math notranslate nohighlight">\(X^T X\)</span> (which is n_features x
n_features), and lam * I adds <span class="math notranslate nohighlight">\(\lambda\)</span> to the diagonal of <span class="math notranslate nohighlight">\(X^T X. We
then invert this matrix and multiply by \)</span>X^T y. The result
for <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is a NumPy array of shape (n_features,) containing the
fitted weights.</p>
<p>This computes the Ridge and OLS regression coefficients directly. The identity
matrix <span class="math notranslate nohighlight">\(I\)</span> has the same size as <span class="math notranslate nohighlight">\(X^T X\)</span>. It adds <span class="math notranslate nohighlight">\(\lambda\)</span> to the diagonal of <span class="math notranslate nohighlight">\(X^T X for Ridge regression. We
then invert this matrix and multiply by \)</span>X^T y<span class="math notranslate nohighlight">\(. The result
for \)</span>\boldsymbol{\theta}<span class="math notranslate nohighlight">\( is a NumPy array of shape (n\)</span>_<span class="math notranslate nohighlight">\(features,) containing the
fitted parameters \)</span>\boldsymbol{\theta}$..</p>
<section id="id1">
<h3>2a)<a class="headerlink" href="#id1" title="Link to this heading">#</a></h3>
<p>Finalize the OLS and Ridge regression determination of the optimal parameters <span class="math notranslate nohighlight">\(bm{\theta}\)</span>.</p>
<p>Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
</section>
<section id="b">
<h3>2b)<a class="headerlink" href="#b" title="Link to this heading">#</a></h3>
<p>Explore the results as function of different values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>. See for example exercise 4 from week 36.</p>
</section>
</section>
<section id="implementing-the-simplest-form-for-gradient-descent">
<h2>Implementing the simplest form for gradient descent<a class="headerlink" href="#implementing-the-simplest-form-for-gradient-descent" title="Link to this heading">#</a></h2>
<section id="exercise-3-implementing-the-simplest-form-for-gradient-descent">
<h2>Exercise 3, Implementing the simplest form for gradient descent<a class="headerlink" href="#exercise-3-implementing-the-simplest-form-for-gradient-descent" title="Link to this heading">#</a></h2>
<p>Alternatively, we can fit the ridge regression model using gradient
descent. This is useful to visualize the iterative convergence and is
necessary if <span class="math notranslate nohighlight">\(n\)</span> and <span class="math notranslate nohighlight">\(p\)</span> are so large that the closed-form might be
@@ -588,16 +588,16 @@ print(&quot;Gradient Descent Ridge coefficients:&quot;, theta_gdRidge)
</div>
<section id="id2">
<h3>3a)<a class="headerlink" href="#id2" title="Link to this heading">#</a></h3>
<p>Discuss the results as function of the learning rate paramaters and the number of iterations.</p>
<p>Discuss the results as function of the learning rate parameters and the number of iterations.</p>
</section>
<section id="id3">
<h3>3b)<a class="headerlink" href="#id3" title="Link to this heading">#</a></h3>
<p>Add a stopping parameter as function of the number iterations.</p>
<p>Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?</p>
<p>If everything worked correctly, the learned coefficients should be
close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to
generate the data. Keep in mind that due to regularization and noise,
the learned values will not exactly equal the true ones, but they
should be in the same ballpark.</p>
should be in the same ballpark. Which method (OLS or Ridge) gives the best results?</p>
</section>
</section>
</section>
@@ -663,12 +663,12 @@ should be in the same ballpark.</p>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#a">1a)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">2a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">2b)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#implementing-the-simplest-form-for-gradient-descent">Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-implementing-the-simplest-form-for-gradient-descent">Exercise 3, Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">3b)</a></li>
</ul>