lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 36

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-03 13:40:18 +02:00
parent 1a1618aa74
commit abd1d73a8f
15 changed files with 668 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb
+30 -29
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "1b638450",
"id": "1b941c35",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "60060788",
"id": "dc05b096",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "7cdd88e4",
"id": "2cf07405",
"metadata": {
"editable": true
},
@@ -37,7 +37,7 @@
"After having completed these exercises you will have:\n",
"1. Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression\n",
"\n",
"2. Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach\n",
"2. Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach\n",
"\n",
"3. Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\\lambda$ in Ridge regression\n",
"\n",
@@ -46,7 +46,7 @@
},
{
"cell_type": "markdown",
"id": "0328fa2a",
"id": "3c139edb",
"metadata": {
"editable": true
},
@@ -58,7 +58,7 @@
},
{
"cell_type": "markdown",
"id": "ac760265",
"id": "aad4cfac",
"metadata": {
"editable": true
},
@@ -70,7 +70,7 @@
},
{
"cell_type": "markdown",
"id": "8d4b0753",
"id": "6682282f",
"metadata": {
"editable": true
},
@@ -83,7 +83,7 @@
},
{
"cell_type": "markdown",
"id": "4517d311",
"id": "89e2f4c4",
"metadata": {
"editable": true
},
@@ -99,7 +99,7 @@
},
{
"cell_type": "markdown",
"id": "da1834a9",
"id": "b06d4e53",
"metadata": {
"editable": true
},
@@ -120,7 +120,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "590b2fb0",
"id": "63796480",
"metadata": {
"collapsed": false,
"editable": true
@@ -140,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "24dd92fc",
"id": "80748600",
"metadata": {
"editable": true
},
@@ -156,7 +156,7 @@
},
{
"cell_type": "markdown",
"id": "4ba80e16",
"id": "92751e5f",
"metadata": {
"editable": true
},
@@ -168,18 +168,18 @@
},
{
"cell_type": "markdown",
"id": "be65f56f",
"id": "aedfbd7a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c265677e",
"id": "5d1288fa",
"metadata": {
"collapsed": false,
"editable": true
@@ -200,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "b989efb9",
"id": "628f5e89",
"metadata": {
"editable": true
},
@@ -214,7 +214,7 @@
},
{
"cell_type": "markdown",
"id": "17b08af7",
"id": "f115ba4e",
"metadata": {
"editable": true
},
@@ -226,7 +226,7 @@
},
{
"cell_type": "markdown",
"id": "6acd4708",
"id": "a9b5189c",
"metadata": {
"editable": true
},
@@ -238,7 +238,7 @@
},
{
"cell_type": "markdown",
"id": "50579422",
"id": "a3969ff6",
"metadata": {
"editable": true
},
@@ -258,7 +258,7 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "c57cc917",
"id": "34d87303",
"metadata": {
"collapsed": false,
"editable": true
@@ -277,13 +277,14 @@
"cost_history = np.zeros(num_iters)\n",
"\n",
"# Gradient descent loop\n",
"m = n_samples # number of examples\n",
"m = n_samples # number of data points\n",
"for t in range(num_iters):\n",
" # Compute prediction error\n",
" error = X_norm.dot(theta) - y_centered \n",
" # Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring\n",
" cost_OLS = ?\n",
" cost_Ridge = ?\n",
" # You could add a history for both methods (optional)\n",
" cost_history[t] = ?\n",
" # Compute gradients for OSL and Ridge\n",
" grad_OLS = ?\n",
@@ -301,7 +302,7 @@
},
{
"cell_type": "markdown",
"id": "e2654903",
"id": "989f70bb",
"metadata": {
"editable": true
},
@@ -313,19 +314,19 @@
},
{
"cell_type": "markdown",
"id": "7e7e7de6",
"id": "370b2dad",
"metadata": {
"editable": true
},
"source": [
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
"Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\\theta$ values. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "75542708",
"id": "ef197cd7",
"metadata": {
"editable": true
},
@@ -351,7 +352,7 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "06077986",
"id": "4ccc2f65",
"metadata": {
"collapsed": false,
"editable": true
@@ -380,7 +381,7 @@
},
{
"cell_type": "markdown",
"id": "86d46505",
"id": "00e279ef",
"metadata": {
"editable": true
},
@@ -394,7 +395,7 @@
},
{
"cell_type": "markdown",
"id": "f1d2f848",
"id": "c910b3f4",
"metadata": {
"editable": true
},
@@ -406,7 +407,7 @@
},
{
"cell_type": "markdown",
"id": "f1c91e84",
"id": "89e6e040",
"metadata": {
"editable": true
},
doc/LectureNotes/_build/html/exercisesweek37.html
+8 -7
View File
@@ -389,7 +389,7 @@ document.write(`
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>
@@ -422,7 +422,7 @@ doconce format html exercisesweek37.do.txt -->
<p>After having completed these exercises you will have:</p>
<ol class="arabic simple">
<li><p>Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression</p></li>
<li><p>Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach</p></li>
<li><p>Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach</p></li>
<li><p>Explore the role of the learning rate in the gradient descent approach and the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span> in Ridge regression</p></li>
<li><p>Scale the data properly</p></li>
</ol>
@@ -482,8 +482,8 @@ same scale).</p>
<h2>Exercise 2, calculate the gradients<a class="headerlink" href="#exercise-2-calculate-the-gradients" title="Link to this heading">#</a></h2>
<p>Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function.</p>
</section>
<section id="exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<section id="exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Set regularization parameter, either a single value or a vector of values
@@ -537,13 +537,14 @@ theta = np.zeros(n_features)
cost_history = np.zeros(num_iters)
# Gradient descent loop
m = n_samples # number of examples
m = n_samples # number of data points
for t in range(num_iters):
# Compute prediction error
error = X_norm.dot(theta) - y_centered
# Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
cost_OLS = ?
cost_Ridge = ?
# You could add a history for both methods (optional)
cost_history[t] = ?
# Compute gradients for OSL and Ridge
grad_OLS = ?
@@ -567,7 +568,7 @@ print(&quot;Gradient Descent Ridge coefficients:&quot;, theta_gdRidge)
</section>
<section id="id3">
<h3>4b)<a class="headerlink" href="#id3" title="Link to this heading">#</a></h3>
<p>Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?</p>
<p>Try to add a stopping parameter as function of the number iterations and the difference between the new and old <span class="math notranslate nohighlight">\(\theta\)</span> values. How would you define a stopping criterion?</p>
</section>
</section>
<section id="exercise-5-ridge-regression-and-a-new-synthetic-dataset">
@@ -697,7 +698,7 @@ should be in the same ballpark. Which method (OLS or Ridge) gives the best resu
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long