lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 36

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-03 13:40:18 +02:00
parent 1a1618aa74
commit abd1d73a8f
15 changed files with 668 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/exercisesweek37.html
+8 -7
View File
@@ -389,7 +389,7 @@ document.write(`
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>
@@ -422,7 +422,7 @@ doconce format html exercisesweek37.do.txt -->
<p>After having completed these exercises you will have:</p>
<ol class="arabic simple">
<li><p>Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression</p></li>
<li><p>Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach</p></li>
<li><p>Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach</p></li>
<li><p>Explore the role of the learning rate in the gradient descent approach and the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span> in Ridge regression</p></li>
<li><p>Scale the data properly</p></li>
</ol>
@@ -482,8 +482,8 @@ same scale).</p>
<h2>Exercise 2, calculate the gradients<a class="headerlink" href="#exercise-2-calculate-the-gradients" title="Link to this heading">#</a></h2>
<p>Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function.</p>
</section>
<section id="exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<section id="exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Set regularization parameter, either a single value or a vector of values
@@ -537,13 +537,14 @@ theta = np.zeros(n_features)
cost_history = np.zeros(num_iters)
# Gradient descent loop
m = n_samples # number of examples
m = n_samples # number of data points
for t in range(num_iters):
# Compute prediction error
error = X_norm.dot(theta) - y_centered
# Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
cost_OLS = ?
cost_Ridge = ?
# You could add a history for both methods (optional)
cost_history[t] = ?
# Compute gradients for OSL and Ridge
grad_OLS = ?
@@ -567,7 +568,7 @@ print(&quot;Gradient Descent Ridge coefficients:&quot;, theta_gdRidge)
</section>
<section id="id3">
<h3>4b)<a class="headerlink" href="#id3" title="Link to this heading">#</a></h3>
<p>Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?</p>
<p>Try to add a stopping parameter as function of the number iterations and the difference between the new and old <span class="math notranslate nohighlight">\(\theta\)</span> values. How would you define a stopping criterion?</p>
</section>
</section>
<section id="exercise-5-ridge-regression-and-a-new-synthetic-dataset">
@@ -697,7 +698,7 @@ should be in the same ballpark. Which method (OLS or Ridge) gives the best resu
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>