lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

addex exercise week 37

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-01 13:43:58 +02:00
parent dccb74ba38
commit eb31c10d1c
41 changed files with 718 additions and 549 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/exercisesweek35.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/exercisesweek37.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_sources/exercisesweek35.ipynb
+1 -1
View File
@@ -323,7 +323,7 @@
"source": [
"n = 100\n",
"x = np.linspace(-3, 3, n)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)"
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 1.0)"
]
},
{
doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb
+146 -94
View File
@@ -2,24 +2,24 @@
"cells": [
{
"cell_type": "markdown",
"id": "d3aa801d",
"id": "b0268cb1",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek37.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 36 -->"
"<!-- dom:TITLE: Exercises week 37 -->"
]
},
{
"cell_type": "markdown",
"id": "7c64e6da",
"id": "700a1d0b",
"metadata": {
"editable": true
},
"source": [
"# Exercises week 36\n",
"# Exercises week 37\n",
"**Implementing gradient descent for Ridge and ordinary Least Squares Regression**\n",
"\n",
"Date: **September 8-12, 2025**"
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "51e35698",
"id": "dbe5809a",
"metadata": {
"editable": true
},
@@ -46,98 +46,44 @@
},
{
"cell_type": "markdown",
"id": "74fb184e",
"id": "ac99e9c0",
"metadata": {
"editable": true
},
"source": [
"## Ridge regression and a new Synthetic Dataset\n",
"## Simple one-dimensional second-order polynomial\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we’ll\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9e6acfef",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
"We start with a very simple function"
]
},
{
"cell_type": "markdown",
"id": "f2d03ca8",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d2d64f9b",
"id": "6d71a32d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"\\f(x)= 2-x+5x^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b4248e9d",
"id": "c6496768",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to."
"defined for $x\\in [-2,2]$. You can add noise if you wish. \n",
"\n",
"We are going to fit this function with a polynomial ansatz. The easiest thing is to set up a second-order polynomial and see if you can fit the above function.\n",
"Feel free to play around with higher-order polynomials."
]
},
{
"cell_type": "markdown",
"id": "5fed181f",
"id": "24678181",
"metadata": {
"editable": true
},
@@ -153,27 +99,28 @@
},
{
"cell_type": "markdown",
"id": "6ec0227c",
"id": "6b1bd90a",
"metadata": {
"editable": true
},
"source": [
"### 1a)\n",
"\n",
"Compute the mean and standard deviation of each column (feature) in $\\boldsymbol{X}$.\n",
"Compute the mean and standard deviation of each column (feature) in your design/feature matrix $\\boldsymbol{X}$.\n",
"Subtract the mean and divide by the standard deviation for each feature.\n",
"\n",
"We will also center the target $\\boldsymbol{y}$ to mean $0$. Centering $\\boldsymbol{y}$\n",
"(and each feature) means the model does not require a separate intercept\n",
"term, the data is shifted such that the intercept is effectively 0\n",
". (In practice, one could include an intercept in the model and not\n",
"penalize it, but here we simplify by centering.)"
"penalize it, but here we simplify by centering.)\n",
"Choose $n=100$ data points and set up $\\boldsymbol{x}, $\\boldsymbol{y} and the design matrix $\\boldsymbol{X}$."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a140aac7",
"execution_count": 1,
"id": "5e751a16",
"metadata": {
"collapsed": false,
"editable": true
@@ -193,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "57ad18f5",
"id": "50a68a52",
"metadata": {
"editable": true
},
@@ -209,18 +156,30 @@
},
{
"cell_type": "markdown",
"id": "2886697d",
"id": "db74a970",
"metadata": {
"editable": true
},
"source": [
"## Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 2, calculate the gradients\n",
"\n",
"Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function."
]
},
{
"cell_type": "markdown",
"id": "f8feaa49",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "97ac6cb6",
"execution_count": 2,
"id": "e6eca7a7",
"metadata": {
"collapsed": false,
"editable": true
@@ -241,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "3efb067b",
"id": "a2dff0cf",
"metadata": {
"editable": true
},
@@ -255,36 +214,36 @@
},
{
"cell_type": "markdown",
"id": "53be2bf8",
"id": "d7542c3e",
"metadata": {
"editable": true
},
"source": [
"### 2a)\n",
"### 3a)\n",
"\n",
"Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters $\\boldsymbol{\\theta}$."
]
},
{
"cell_type": "markdown",
"id": "e4126591",
"id": "b922fd54",
"metadata": {
"editable": true
},
"source": [
"### 2b)\n",
"### 3b)\n",
"\n",
"Explore the results as function of different values of the hyperparameter $\\lambda$. See for example exercise 4 from week 36."
]
},
{
"cell_type": "markdown",
"id": "642d0850",
"id": "92268a9a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, Implementing the simplest form for gradient descent\n",
"## Exercise 4, Implementing the simplest form for gradient descent\n",
"\n",
"Alternatively, we can fit the ridge regression model using gradient\n",
"descent. This is useful to visualize the iterative convergence and is\n",
@@ -298,8 +257,8 @@
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a67af634",
"execution_count": 3,
"id": "81660fa3",
"metadata": {
"collapsed": false,
"editable": true
@@ -342,26 +301,119 @@
},
{
"cell_type": "markdown",
"id": "1c8c35dc",
"id": "95149551",
"metadata": {
"editable": true
},
"source": [
"### 3a)\n",
"### 4a)\n",
"\n",
"Discuss the results as function of the learning rate parameters and the number of iterations."
]
},
{
"cell_type": "markdown",
"id": "899fec5c",
"id": "09b5400e",
"metadata": {
"editable": true
},
"source": [
"### 3b)\n",
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion? \n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "c635ca9e",
"metadata": {
"editable": true
},
"source": [
"## Exercise 5, Ridge regression and a new Synthetic Dataset\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we’ll\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4ca4ce05",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
]
},
{
"cell_type": "markdown",
"id": "af39d8bc",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d35d3438",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b28bf122",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to. \n",
"\n",
"Try to fit the above data set using OLS and Ridge regression with the analytical expressions and your own gradient descent codes.\n",
"\n",
"If everything worked correctly, the learned coefficients should be\n",
"close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to\n",
doc/LectureNotes/_build/html/chapter1.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter10.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter11.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter12.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter13.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter2.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter3.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter4.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter5.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter6.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter7.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter8.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapter9.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/chapteroptimization.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/clustering.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/exercisesweek34.html
+1 -1
View File
@@ -227,7 +227,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/exercisesweek35.html
+2 -2
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
@@ -553,7 +553,7 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">)</span>
</pre></div>
</div>
</div>
doc/LectureNotes/_build/html/exercisesweek36.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/exercisesweek37.html
+95 -74
View File
@@ -8,7 +8,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Exercises week 36 &#8212; Applied Data Analysis and Machine Learning</title>
<title>Exercises week 37 &#8212; Applied Data Analysis and Machine Learning</title>
@@ -228,7 +228,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Exercises week 36</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Exercises week 37</a></li>
</ul>
</div>
@@ -367,7 +367,7 @@ document.write(`
<div id="jb-print-docs-body" class="onlyprint">
<h1>Exercises week 36</h1>
<h1>Exercises week 37</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
@@ -378,21 +378,23 @@ document.write(`
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-goals">Learning goals</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ridge-regression-and-a-new-synthetic-dataset">Ridge regression and a new Synthetic Dataset</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-one-dimensional-second-order-polynomial">Simple one-dimensional second-order polynomial</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-scale-your-data">Exercise 1, scale your data</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#a">1a)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">2a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">2b)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-implementing-the-simplest-form-for-gradient-descent">Exercise 3, Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">3b)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-implementing-the-simplest-form-for-gradient-descent">Exercise 4, Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">4a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">4b)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-ridge-regression-and-a-new-synthetic-dataset">Exercise 5, Ridge regression and a new Synthetic Dataset</a></li>
</ul>
</nav>
</div>
@@ -406,8 +408,8 @@ document.write(`
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html exercisesweek37.do.txt -->
<!-- dom:TITLE: Exercises week 36 --><section class="tex2jax_ignore mathjax_ignore" id="exercises-week-36">
<h1>Exercises week 36<a class="headerlink" href="#exercises-week-36" title="Link to this heading">#</a></h1>
<!-- dom:TITLE: Exercises week 37 --><section class="tex2jax_ignore mathjax_ignore" id="exercises-week-37">
<h1>Exercises week 37<a class="headerlink" href="#exercises-week-37" title="Link to this heading">#</a></h1>
<p><strong>Implementing gradient descent for Ridge and ordinary Least Squares Regression</strong></p>
<p>Date: <strong>September 8-12, 2025</strong></p>
<section id="learning-goals">
@@ -420,53 +422,16 @@ doconce format html exercisesweek37.do.txt -->
<li><p>Scale the data properly</p></li>
</ol>
</section>
<section id="ridge-regression-and-a-new-synthetic-dataset">
<h2>Ridge regression and a new Synthetic Dataset<a class="headerlink" href="#ridge-regression-and-a-new-synthetic-dataset" title="Link to this heading">#</a></h2>
<p>We create a synthetic linear regression dataset with a sparse
underlying relationship. This means we have many features but only a
few of them actually contribute to the target. In our example, we’ll
use 10 features with only 3 non-zero weights in the true model. This
way, the target is generated as a linear combination of a few features
(with known coefficients) plus some random noise. The steps we include are:</p>
<p>Decide on the number of samples and features (e.g. 100 samples, 10 features).
Define the <strong>true</strong> coefficient vector with mostly zeros (for sparsity). For example, we set <span class="math notranslate nohighlight">\(\hat{\boldsymbol{\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]\)</span>, meaning only features 0, 1, and 6 have a real effect on y.</p>
<p>Then we sample feature values for <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.
Then we compute the target values <span class="math notranslate nohighlight">\(y\)</span> using the linear combination <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span> and add some noise (to simulate measurement error or unexplained variance).</p>
<p>Below is the code to generate the dataset:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Define dataset size
n_samples = 100
n_features = 10
# Define true coefficients (sparse linear relationship)
theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])
# Generate feature matrix X (n_samples x n_features) with random values
X = np.random.randn(n_samples, n_features) # standard normal distribution
# Generate target values y with a linear combination of X and theta_true, plus noise
noise = 0.5 * np.random.randn(n_samples) # Gaussian noise
y = X.dot @ theta_true + noise
</pre></div>
</div>
</div>
</div>
<p>This code produces a dataset where only features 0, 1, and 6
significantly influence <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. The rest of the features have zero true
coefficient. For example, feature 0 has
a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so
the expected relationship is:</p>
<section id="simple-one-dimensional-second-order-polynomial">
<h2>Simple one-dimensional second-order polynomial<a class="headerlink" href="#simple-one-dimensional-second-order-polynomial" title="Link to this heading">#</a></h2>
<p>We start with a very simple function</p>
<div class="math notranslate nohighlight">
\[
y \approx 5 \times x_0 \;-\; 3 \times x_1 \;+\; 2 \times x_6 \;+\; \text{noise}.
\f(x)= 2-x+5x^2,
\]</div>
<p>You can remove the noise if you wish to.</p>
<p>defined for <span class="math notranslate nohighlight">\(x\in [-2,2]\)</span>. You can add noise if you wish.</p>
<p>We are going to fit this function with a polynomial ansatz. The easiest thing is to set up a second-order polynomial and see if you can fit the above function.
Feel free to play around with higher-order polynomials.</p>
</section>
<section id="exercise-1-scale-your-data">
<h2>Exercise 1, scale your data<a class="headerlink" href="#exercise-1-scale-your-data" title="Link to this heading">#</a></h2>
@@ -477,13 +442,14 @@ regularization. Here we will perform standardization, scaling each
feature to have mean 0 and standard deviation 1.</p>
<section id="a">
<h3>1a)<a class="headerlink" href="#a" title="Link to this heading">#</a></h3>
<p>Compute the mean and standard deviation of each column (feature) in <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.
<p>Compute the mean and standard deviation of each column (feature) in your design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.
Subtract the mean and divide by the standard deviation for each feature.</p>
<p>We will also center the target <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> to mean <span class="math notranslate nohighlight">\(0\)</span>. Centering <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>
(and each feature) means the model does not require a separate intercept
term, the data is shifted such that the intercept is effectively 0
. (In practice, one could include an intercept in the model and not
penalize it, but here we simplify by centering.)</p>
penalize it, but here we simplify by centering.)
Choose <span class="math notranslate nohighlight">\(n=100\)</span> data points and set up <span class="math notranslate nohighlight">\(\boldsymbol{x}, \)</span>\boldsymbol{y} and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Standardize features (zero mean, unit variance for each feature)
@@ -507,8 +473,12 @@ nicer and ensures the regularization penalty <span class="math notranslate nohig
same scale).</p>
</section>
</section>
<section id="exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<section id="exercise-2-calculate-the-gradients">
<h2>Exercise 2, calculate the gradients<a class="headerlink" href="#exercise-2-calculate-the-gradients" title="Link to this heading">#</a></h2>
<p>Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function.</p>
</section>
<section id="exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Set regularization parameter, either a single value or a vector of values
@@ -531,16 +501,16 @@ then invert this matrix and multiply by \)</span>X^T y<span class="math notransl
for \)</span>\boldsymbol{\theta}<span class="math notranslate nohighlight">\( is a NumPy array of shape (n\)</span>_<span class="math notranslate nohighlight">\(features,) containing the
fitted parameters \)</span>\boldsymbol{\theta}$..</p>
<section id="id1">
<h3>2a)<a class="headerlink" href="#id1" title="Link to this heading">#</a></h3>
<h3>3a)<a class="headerlink" href="#id1" title="Link to this heading">#</a></h3>
<p>Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
</section>
<section id="b">
<h3>2b)<a class="headerlink" href="#b" title="Link to this heading">#</a></h3>
<h3>3b)<a class="headerlink" href="#b" title="Link to this heading">#</a></h3>
<p>Explore the results as function of different values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>. See for example exercise 4 from week 36.</p>
</section>
</section>
<section id="exercise-3-implementing-the-simplest-form-for-gradient-descent">
<h2>Exercise 3, Implementing the simplest form for gradient descent<a class="headerlink" href="#exercise-3-implementing-the-simplest-form-for-gradient-descent" title="Link to this heading">#</a></h2>
<section id="exercise-4-implementing-the-simplest-form-for-gradient-descent">
<h2>Exercise 4, Implementing the simplest form for gradient descent<a class="headerlink" href="#exercise-4-implementing-the-simplest-form-for-gradient-descent" title="Link to this heading">#</a></h2>
<p>Alternatively, we can fit the ridge regression model using gradient
descent. This is useful to visualize the iterative convergence and is
necessary if <span class="math notranslate nohighlight">\(n\)</span> and <span class="math notranslate nohighlight">\(p\)</span> are so large that the closed-form might be
@@ -587,19 +557,68 @@ print(&quot;Gradient Descent Ridge coefficients:&quot;, theta_gdRidge)
</div>
</div>
<section id="id2">
<h3>3a)<a class="headerlink" href="#id2" title="Link to this heading">#</a></h3>
<h3>4a)<a class="headerlink" href="#id2" title="Link to this heading">#</a></h3>
<p>Discuss the results as function of the learning rate parameters and the number of iterations.</p>
</section>
<section id="id3">
<h3>3b)<a class="headerlink" href="#id3" title="Link to this heading">#</a></h3>
<h3>4b)<a class="headerlink" href="#id3" title="Link to this heading">#</a></h3>
<p>Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?</p>
</section>
</section>
<section id="exercise-5-ridge-regression-and-a-new-synthetic-dataset">
<h2>Exercise 5, Ridge regression and a new Synthetic Dataset<a class="headerlink" href="#exercise-5-ridge-regression-and-a-new-synthetic-dataset" title="Link to this heading">#</a></h2>
<p>We create a synthetic linear regression dataset with a sparse
underlying relationship. This means we have many features but only a
few of them actually contribute to the target. In our example, we’ll
use 10 features with only 3 non-zero weights in the true model. This
way, the target is generated as a linear combination of a few features
(with known coefficients) plus some random noise. The steps we include are:</p>
<p>Decide on the number of samples and features (e.g. 100 samples, 10 features).
Define the <strong>true</strong> coefficient vector with mostly zeros (for sparsity). For example, we set <span class="math notranslate nohighlight">\(\hat{\boldsymbol{\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]\)</span>, meaning only features 0, 1, and 6 have a real effect on y.</p>
<p>Then we sample feature values for <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.
Then we compute the target values <span class="math notranslate nohighlight">\(y\)</span> using the linear combination <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span> and add some noise (to simulate measurement error or unexplained variance).</p>
<p>Below is the code to generate the dataset:</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Define dataset size
n_samples = 100
n_features = 10
# Define true coefficients (sparse linear relationship)
theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])
# Generate feature matrix X (n_samples x n_features) with random values
X = np.random.randn(n_samples, n_features) # standard normal distribution
# Generate target values y with a linear combination of X and theta_true, plus noise
noise = 0.5 * np.random.randn(n_samples) # Gaussian noise
y = X.dot @ theta_true + noise
</pre></div>
</div>
</div>
</div>
<p>This code produces a dataset where only features 0, 1, and 6
significantly influence <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. The rest of the features have zero true
coefficient. For example, feature 0 has
a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so
the expected relationship is:</p>
<div class="math notranslate nohighlight">
\[
y \approx 5 \times x_0 \;-\; 3 \times x_1 \;+\; 2 \times x_6 \;+\; \text{noise}.
\]</div>
<p>You can remove the noise if you wish to.</p>
<p>Try to fit the above data set using OLS and Ridge regression with the analytical expressions and your own gradient descent codes.</p>
<p>If everything worked correctly, the learned coefficients should be
close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to
generate the data. Keep in mind that due to regularization and noise,
the learned values will not exactly equal the true ones, but they
should be in the same ballpark. Which method (OLS or Ridge) gives the best results?</p>
</section>
</section>
</section>
<script type="text/x-thebe-config">
@@ -658,21 +677,23 @@ should be in the same ballpark. Which method (OLS or Ridge) gives the best resu
<nav class="bd-toc-nav page-toc">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-goals">Learning goals</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ridge-regression-and-a-new-synthetic-dataset">Ridge regression and a new Synthetic Dataset</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-one-dimensional-second-order-polynomial">Simple one-dimensional second-order polynomial</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-scale-your-data">Exercise 1, scale your data</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#a">1a)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">2a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">2b)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-implementing-the-simplest-form-for-gradient-descent">Exercise 3, Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">3b)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-implementing-the-simplest-form-for-gradient-descent">Exercise 4, Implementing the simplest form for gradient descent</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">4a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">4b)</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-ridge-regression-and-a-new-synthetic-dataset">Exercise 5, Ridge regression and a new Synthetic Dataset</a></li>
</ul>
</nav></div>
doc/LectureNotes/_build/html/genindex.html
+1 -1
View File
@@ -226,7 +226,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/intro.html
+1 -1
View File
@@ -230,7 +230,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/linalg.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/objects.inv
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/schedule.html
+1 -1
View File
@@ -227,7 +227,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/search.html
+1 -1
View File
@@ -228,7 +228,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/html/statistics.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/teachers.html
+1 -1
View File
@@ -227,7 +227,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/textbooks.html
+1 -1
View File
@@ -227,7 +227,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/week34.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/week35.html
+1 -1
View File
@@ -229,7 +229,7 @@
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
doc/LectureNotes/_build/html/week36.html
+3 -3
View File
@@ -62,7 +62,7 @@
<script>DOCUMENTATION_OPTIONS.pagename = 'week36';</script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Exercises week 36" href="exercisesweek37.html" />
<link rel="next" title="Exercises week 37" href="exercisesweek37.html" />
<link rel="prev" title="Exercises week 36" href="exercisesweek36.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
@@ -229,7 +229,7 @@
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
</ul>
</div>
@@ -2410,7 +2410,7 @@ plt.show()
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Exercises week 36</p>
<p class="prev-next-title">Exercises week 37</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
doc/LectureNotes/_build/jupyter_execute/exercisesweek35.ipynb
+1 -1
View File
@@ -323,7 +323,7 @@
"source": [
"n = 100\n",
"x = np.linspace(-3, 3, n)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)"
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 1.0)"
]
},
{
doc/LectureNotes/_build/jupyter_execute/exercisesweek37.ipynb
+146 -94
View File
@@ -2,24 +2,24 @@
"cells": [
{
"cell_type": "markdown",
"id": "d3aa801d",
"id": "b0268cb1",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek37.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 36 -->"
"<!-- dom:TITLE: Exercises week 37 -->"
]
},
{
"cell_type": "markdown",
"id": "7c64e6da",
"id": "700a1d0b",
"metadata": {
"editable": true
},
"source": [
"# Exercises week 36\n",
"# Exercises week 37\n",
"**Implementing gradient descent for Ridge and ordinary Least Squares Regression**\n",
"\n",
"Date: **September 8-12, 2025**"
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "51e35698",
"id": "dbe5809a",
"metadata": {
"editable": true
},
@@ -46,98 +46,44 @@
},
{
"cell_type": "markdown",
"id": "74fb184e",
"id": "ac99e9c0",
"metadata": {
"editable": true
},
"source": [
"## Ridge regression and a new Synthetic Dataset\n",
"## Simple one-dimensional second-order polynomial\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we’ll\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9e6acfef",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
"We start with a very simple function"
]
},
{
"cell_type": "markdown",
"id": "f2d03ca8",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d2d64f9b",
"id": "6d71a32d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"\\f(x)= 2-x+5x^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b4248e9d",
"id": "c6496768",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to."
"defined for $x\\in [-2,2]$. You can add noise if you wish. \n",
"\n",
"We are going to fit this function with a polynomial ansatz. The easiest thing is to set up a second-order polynomial and see if you can fit the above function.\n",
"Feel free to play around with higher-order polynomials."
]
},
{
"cell_type": "markdown",
"id": "5fed181f",
"id": "24678181",
"metadata": {
"editable": true
},
@@ -153,27 +99,28 @@
},
{
"cell_type": "markdown",
"id": "6ec0227c",
"id": "6b1bd90a",
"metadata": {
"editable": true
},
"source": [
"### 1a)\n",
"\n",
"Compute the mean and standard deviation of each column (feature) in $\\boldsymbol{X}$.\n",
"Compute the mean and standard deviation of each column (feature) in your design/feature matrix $\\boldsymbol{X}$.\n",
"Subtract the mean and divide by the standard deviation for each feature.\n",
"\n",
"We will also center the target $\\boldsymbol{y}$ to mean $0$. Centering $\\boldsymbol{y}$\n",
"(and each feature) means the model does not require a separate intercept\n",
"term, the data is shifted such that the intercept is effectively 0\n",
". (In practice, one could include an intercept in the model and not\n",
"penalize it, but here we simplify by centering.)"
"penalize it, but here we simplify by centering.)\n",
"Choose $n=100$ data points and set up $\\boldsymbol{x}, $\\boldsymbol{y} and the design matrix $\\boldsymbol{X}$."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a140aac7",
"execution_count": 1,
"id": "5e751a16",
"metadata": {
"collapsed": false,
"editable": true
@@ -193,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "57ad18f5",
"id": "50a68a52",
"metadata": {
"editable": true
},
@@ -209,18 +156,30 @@
},
{
"cell_type": "markdown",
"id": "2886697d",
"id": "db74a970",
"metadata": {
"editable": true
},
"source": [
"## Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 2, calculate the gradients\n",
"\n",
"Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function."
]
},
{
"cell_type": "markdown",
"id": "f8feaa49",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "97ac6cb6",
"execution_count": 2,
"id": "e6eca7a7",
"metadata": {
"collapsed": false,
"editable": true
@@ -241,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "3efb067b",
"id": "a2dff0cf",
"metadata": {
"editable": true
},
@@ -255,36 +214,36 @@
},
{
"cell_type": "markdown",
"id": "53be2bf8",
"id": "d7542c3e",
"metadata": {
"editable": true
},
"source": [
"### 2a)\n",
"### 3a)\n",
"\n",
"Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters $\\boldsymbol{\\theta}$."
]
},
{
"cell_type": "markdown",
"id": "e4126591",
"id": "b922fd54",
"metadata": {
"editable": true
},
"source": [
"### 2b)\n",
"### 3b)\n",
"\n",
"Explore the results as function of different values of the hyperparameter $\\lambda$. See for example exercise 4 from week 36."
]
},
{
"cell_type": "markdown",
"id": "642d0850",
"id": "92268a9a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, Implementing the simplest form for gradient descent\n",
"## Exercise 4, Implementing the simplest form for gradient descent\n",
"\n",
"Alternatively, we can fit the ridge regression model using gradient\n",
"descent. This is useful to visualize the iterative convergence and is\n",
@@ -298,8 +257,8 @@
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a67af634",
"execution_count": 3,
"id": "81660fa3",
"metadata": {
"collapsed": false,
"editable": true
@@ -342,26 +301,119 @@
},
{
"cell_type": "markdown",
"id": "1c8c35dc",
"id": "95149551",
"metadata": {
"editable": true
},
"source": [
"### 3a)\n",
"### 4a)\n",
"\n",
"Discuss the results as function of the learning rate parameters and the number of iterations."
]
},
{
"cell_type": "markdown",
"id": "899fec5c",
"id": "09b5400e",
"metadata": {
"editable": true
},
"source": [
"### 3b)\n",
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion? \n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "c635ca9e",
"metadata": {
"editable": true
},
"source": [
"## Exercise 5, Ridge regression and a new Synthetic Dataset\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we’ll\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4ca4ce05",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
]
},
{
"cell_type": "markdown",
"id": "af39d8bc",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d35d3438",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b28bf122",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to. \n",
"\n",
"Try to fit the above data set using OLS and Ridge regression with the analytical expressions and your own gradient descent codes.\n",
"\n",
"If everything worked correctly, the learned coefficients should be\n",
"close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to\n",
doc/LectureNotes/exercisesweek37.ipynb
+151 -129
View File
@@ -2,24 +2,24 @@
"cells": [
{
"cell_type": "markdown",
"id": "d3aa801d",
"id": "b0268cb1",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek37.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 36 -->"
"<!-- dom:TITLE: Exercises week 37 -->"
]
},
{
"cell_type": "markdown",
"id": "7c64e6da",
"id": "700a1d0b",
"metadata": {
"editable": true
},
"source": [
"# Exercises week 36\n",
"# Exercises week 37\n",
"**Implementing gradient descent for Ridge and ordinary Least Squares Regression**\n",
"\n",
"Date: **September 8-12, 2025**"
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "51e35698",
"id": "dbe5809a",
"metadata": {
"editable": true
},
@@ -37,7 +37,7 @@
"After having completed these exercises you will have:\n",
"1. Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression\n",
"\n",
"2. Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach\n",
"2. Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach\n",
"\n",
"3. Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\\lambda$ in Ridge regression\n",
"\n",
@@ -46,101 +46,44 @@
},
{
"cell_type": "markdown",
"id": "74fb184e",
"id": "ac99e9c0",
"metadata": {
"editable": true
},
"source": [
"## Ridge regression and a new Synthetic Dataset\n",
"## Simple one-dimensional second-order polynomial\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we will\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9e6acfef",
"metadata": {
"collapsed": false,
"editable": true,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
"We start with a very simple function"
]
},
{
"cell_type": "markdown",
"id": "f2d03ca8",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d2d64f9b",
"id": "6d71a32d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"\\f(x)= 2-x+5x^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b4248e9d",
"id": "c6496768",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to."
"defined for $x\\in [-2,2]$. You can add noise if you wish. \n",
"\n",
"We are going to fit this function with a polynomial ansatz. The easiest thing is to set up a second-order polynomial and see if you can fit the above function.\n",
"Feel free to play around with higher-order polynomials."
]
},
{
"cell_type": "markdown",
"id": "5fed181f",
"id": "24678181",
"metadata": {
"editable": true
},
@@ -156,33 +99,31 @@
},
{
"cell_type": "markdown",
"id": "6ec0227c",
"id": "6b1bd90a",
"metadata": {
"editable": true
},
"source": [
"### 1a)\n",
"\n",
"Compute the mean and standard deviation of each column (feature) in $\\boldsymbol{X}$.\n",
"Compute the mean and standard deviation of each column (feature) in your design/feature matrix $\\boldsymbol{X}$.\n",
"Subtract the mean and divide by the standard deviation for each feature.\n",
"\n",
"We will also center the target $\\boldsymbol{y}$ to mean $0$. Centering $\\boldsymbol{y}$\n",
"(and each feature) means the model does not require a separate intercept\n",
"term, the data is shifted such that the intercept is effectively 0\n",
". (In practice, one could include an intercept in the model and not\n",
"penalize it, but here we simplify by centering.)"
"penalize it, but here we simplify by centering.)\n",
"Choose $n=100$ data points and set up $\\boldsymbol{x}, $\\boldsymbol{y} and the design matrix $\\boldsymbol{X}$."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a140aac7",
"execution_count": 1,
"id": "5e751a16",
"metadata": {
"collapsed": false,
"editable": true,
"jupyter": {
"outputs_hidden": false
}
"editable": true
},
"outputs": [],
"source": [
@@ -199,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "57ad18f5",
"id": "50a68a52",
"metadata": {
"editable": true
},
@@ -215,24 +156,33 @@
},
{
"cell_type": "markdown",
"id": "2886697d",
"id": "db74a970",
"metadata": {
"editable": true
},
"source": [
"## Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 2, calculate the gradients\n",
"\n",
"Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function."
]
},
{
"cell_type": "markdown",
"id": "f8feaa49",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "97ac6cb6",
"execution_count": 2,
"id": "e6eca7a7",
"metadata": {
"collapsed": false,
"editable": true,
"jupyter": {
"outputs_hidden": false
}
"editable": true
},
"outputs": [],
"source": [
@@ -250,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "3efb067b",
"id": "a2dff0cf",
"metadata": {
"editable": true
},
@@ -264,36 +214,36 @@
},
{
"cell_type": "markdown",
"id": "53be2bf8",
"id": "d7542c3e",
"metadata": {
"editable": true
},
"source": [
"### 2a)\n",
"### 3a)\n",
"\n",
"Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters $\\boldsymbol{\\theta}$."
]
},
{
"cell_type": "markdown",
"id": "e4126591",
"id": "b922fd54",
"metadata": {
"editable": true
},
"source": [
"### 2b)\n",
"### 3b)\n",
"\n",
"Explore the results as function of different values of the hyperparameter $\\lambda$. See for example exercise 4 from week 36."
]
},
{
"cell_type": "markdown",
"id": "642d0850",
"id": "92268a9a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, Implementing the simplest form for gradient descent\n",
"## Exercise 4, Implementing the simplest form for gradient descent\n",
"\n",
"Alternatively, we can fit the ridge regression model using gradient\n",
"descent. This is useful to visualize the iterative convergence and is\n",
@@ -307,14 +257,11 @@
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a67af634",
"execution_count": 3,
"id": "81660fa3",
"metadata": {
"collapsed": false,
"editable": true,
"jupyter": {
"outputs_hidden": false
}
"editable": true
},
"outputs": [],
"source": [
@@ -354,26 +301,119 @@
},
{
"cell_type": "markdown",
"id": "1c8c35dc",
"id": "95149551",
"metadata": {
"editable": true
},
"source": [
"### 3a)\n",
"### 4a)\n",
"\n",
"Discuss the results as function of the learning rate parameters and the number of iterations."
]
},
{
"cell_type": "markdown",
"id": "899fec5c",
"id": "09b5400e",
"metadata": {
"editable": true
},
"source": [
"### 3b)\n",
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion? \n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "c635ca9e",
"metadata": {
"editable": true
},
"source": [
"## Exercise 5, Ridge regression and a new Synthetic Dataset\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we’ll\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4ca4ce05",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
]
},
{
"cell_type": "markdown",
"id": "af39d8bc",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d35d3438",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b28bf122",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to. \n",
"\n",
"Try to fit the above data set using OLS and Ridge regression with the analytical expressions and your own gradient descent codes.\n",
"\n",
"If everything worked correctly, the learned coefficients should be\n",
"close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to\n",
@@ -383,25 +423,7 @@
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.15"
}
},
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
doc/src/week37/exercisesweek37.do.txt
+145 -123
View File
@@ -1,4 +1,4 @@
TITLE: Exercises week 36
TITLE: Exercises week 37
AUTHOR: Implementing gradient descent for Ridge and ordinary Least Squares Regression
DATE: September 8-12, 2025
@@ -11,7 +11,149 @@ o Be able to compare the analytical expressions for OLS and Rudge regression wit
o Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\lambda$ in Ridge regression
o Scale the data properly
===== Ridge regression and a new Synthetic Dataset =====
===== Simple one-dimensional second-order polynomial =====
We start with a very simple function
!bt
\[
\f(x)= 2-x+5x^2,
\]
!et
defined for $x\in [-2,2]$. You can add noise if you wish.
We are going to fit this function with a polynomial ansatz. The easiest thing is to set up a second-order polynomial and see if you can fit the above function.
Feel free to play around with higher-order polynomials.
===== Exercise 1, scale your data =====
Before fitting a regression model, it is good practice to normalize or
standardize the features. This ensures all features are on a
comparable scale, which is especially important when using
regularization. Here we will perform standardization, scaling each
feature to have mean 0 and standard deviation 1.
=== 1a) ===
Compute the mean and standard deviation of each column (feature) in your design/feature matrix $\bm{X}$.
Subtract the mean and divide by the standard deviation for each feature.
We will also center the target $\bm{y}$ to mean $0$. Centering $\bm{y}$
(and each feature) means the model does not require a separate intercept
term, the data is shifted such that the intercept is effectively 0
. (In practice, one could include an intercept in the model and not
penalize it, but here we simplify by centering.)
Choose $n=100$ data points and set up $\bm{x}, $\bm{y} and the design matrix $\bm{X}$.
!bc pycod
# Standardize features (zero mean, unit variance for each feature)
X_mean = X.mean(axis=0)
X_std = X.std(axis=0)
X_std[X_std == 0] = 1 # safeguard to avoid division by zero for constant features
X_norm = (X - X_mean) / X_std
# Center the target to zero mean (optional, to simplify intercept handling)
y_mean = ?
y_centered = ?
!ec
Fill in the necessary details.
After this preprocessing, each column of $\bm{X}_{\mathrm{norm}}$ has mean zero and standard deviation $1$
and $\bm{y}_{\mathrm{centered}}$ has mean 0. This makes the optimization landscape
nicer and ensures the regularization penalty $\lambda \sum_j
\theta_j^2$ in Ridge regression treats each coefficient fairly (since features are on the
same scale).
===== Exercise 2, calculate the gradients =====
Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function.
===== Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\bm{\theta}$ =====
!bc pycod
# Set regularization parameter, either a single value or a vector of values
lambda = ?
# Analytical form for OLS and Ridge solution: theta_Ridge = (X^T X + lambda * I)^{-1} X^T y and theta_OLS = (X^T X)^{-1} X^T y
I = np.eye(n_features)
theta_closed_formRidge = ?
theta_closed_formOLS = ?
print("Closed-form Ridge coefficients:", theta_closed_form)
print("Closed-form OLS coefficients:", theta_closed_form)
!ec
This computes the Ridge and OLS regression coefficients directly. The identity
matrix $I$ has the same size as $X^T X$. It adds $\lambda$ to the diagonal of $X^T X for Ridge regression. We
then invert this matrix and multiply by $X^T y$. The result
for $\bm{\theta}$ is a NumPy array of shape (n$\_$features,) containing the
fitted parameters $\bm{\theta}$..
=== 3a) ===
Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters $\bm{\theta}$.
=== 3b) ===
Explore the results as function of different values of the hyperparameter $\lambda$. See for example exercise 4 from week 36.
===== Exercise 4, Implementing the simplest form for gradient descent =====
Alternatively, we can fit the ridge regression model using gradient
descent. This is useful to visualize the iterative convergence and is
necessary if $n$ and $p$ are so large that the closed-form might be
too slow or memory-intensive. We derive the gradients from the cost
functions defined above. Use the gradients of the Ridge and OLS cost functions with respect to
the parameters $\bm{\theta}$ and set up (using the template below) your own gradient descent code for OLS and Ridge regression.
Below is a template code for gradient descent implementation of ridge:
!bc pycod
# Gradient descent parameters, learning rate eta first
eta = 0.1
# Then number of iterations
num_iters = 1000
# Initialize weights for gradient descent
theta = np.zeros(n_features)
# Arrays to store history for plotting
cost_history = np.zeros(num_iters)
# Gradient descent loop
m = n_samples # number of examples
for t in range(num_iters):
# Compute prediction error
error = X_norm.dot(theta) - y_centered
# Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
cost_OLS = ?
cost_Ridge = ?
cost_history[t] = ?
# Compute gradients for OSL and Ridge
grad_OLS = ?
grad_Ridge = ?
# Update parameters theta
theta_gdOLS = ?
theta_gdRidge = ?
# After the loop, theta contains the fitted coefficients
theta_gdOLS = ?
theta_gdRidge = ?
print("Gradient Descent OLS coefficients:", theta_gdOLS)
print("Gradient Descent Ridge coefficients:", theta_gdRidge)
!ec
=== 4a) ===
Discuss the results as function of the learning rate parameters and the number of iterations.
=== 4b) ===
Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?
===== Exercise 5, Ridge regression and a new Synthetic Dataset =====
We create a synthetic linear regression dataset with a sparse
@@ -62,128 +204,8 @@ y \approx 5 \times x_0 \;-\; 3 \times x_1 \;+\; 2 \times x_6 \;+\; \text{noise}.
!et
You can remove the noise if you wish to.
===== Exercise 1, scale your data =====
Before fitting a regression model, it is good practice to normalize or
standardize the features. This ensures all features are on a
comparable scale, which is especially important when using
regularization. Here we will perform standardization, scaling each
feature to have mean 0 and standard deviation 1.
=== 1a) ===
Compute the mean and standard deviation of each column (feature) in $\bm{X}$.
Subtract the mean and divide by the standard deviation for each feature.
We will also center the target $\bm{y}$ to mean $0$. Centering $\bm{y}$
(and each feature) means the model does not require a separate intercept
term, the data is shifted such that the intercept is effectively 0
. (In practice, one could include an intercept in the model and not
penalize it, but here we simplify by centering.)
!bc pycod
# Standardize features (zero mean, unit variance for each feature)
X_mean = X.mean(axis=0)
X_std = X.std(axis=0)
X_std[X_std == 0] = 1 # safeguard to avoid division by zero for constant features
X_norm = (X - X_mean) / X_std
# Center the target to zero mean (optional, to simplify intercept handling)
y_mean = ?
y_centered = ?
!ec
Fill in the necessary details.
After this preprocessing, each column of $\bm{X}_{\mathrm{norm}}$ has mean zero and standard deviation $1$
and $\bm{y}_{\mathrm{centered}}$ has mean 0. This makes the optimization landscape
nicer and ensures the regularization penalty $\lambda \sum_j
\theta_j^2$ in Ridge regression treats each coefficient fairly (since features are on the
same scale).
===== Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\bm{\theta}$ =====
!bc pycod
# Set regularization parameter, either a single value or a vector of values
lambda = ?
# Analytical form for OLS and Ridge solution: theta_Ridge = (X^T X + lambda * I)^{-1} X^T y and theta_OLS = (X^T X)^{-1} X^T y
I = np.eye(n_features)
theta_closed_formRidge = ?
theta_closed_formOLS = ?
print("Closed-form Ridge coefficients:", theta_closed_form)
print("Closed-form OLS coefficients:", theta_closed_form)
!ec
This computes the Ridge and OLS regression coefficients directly. The identity
matrix $I$ has the same size as $X^T X$. It adds $\lambda$ to the diagonal of $X^T X for Ridge regression. We
then invert this matrix and multiply by $X^T y$. The result
for $\bm{\theta}$ is a NumPy array of shape (n$\_$features,) containing the
fitted parameters $\bm{\theta}$..
=== 2a) ===
Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters $\bm{\theta}$.
=== 2b) ===
Explore the results as function of different values of the hyperparameter $\lambda$. See for example exercise 4 from week 36.
===== Exercise 3, Implementing the simplest form for gradient descent =====
Alternatively, we can fit the ridge regression model using gradient
descent. This is useful to visualize the iterative convergence and is
necessary if $n$ and $p$ are so large that the closed-form might be
too slow or memory-intensive. We derive the gradients from the cost
functions defined above. Use the gradients of the Ridge and OLS cost functions with respect to
the parameters $\bm{\theta}$ and set up (using the template below) your own gradient descent code for OLS and Ridge regression.
Below is a template code for gradient descent implementation of ridge:
!bc pycod
# Gradient descent parameters, learning rate eta first
eta = 0.1
# Then number of iterations
num_iters = 1000
# Initialize weights for gradient descent
theta = np.zeros(n_features)
# Arrays to store history for plotting
cost_history = np.zeros(num_iters)
# Gradient descent loop
m = n_samples # number of examples
for t in range(num_iters):
# Compute prediction error
error = X_norm.dot(theta) - y_centered
# Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
cost_OLS = ?
cost_Ridge = ?
cost_history[t] = ?
# Compute gradients for OSL and Ridge
grad_OLS = ?
grad_Ridge = ?
# Update parameters theta
theta_gdOLS = ?
theta_gdRidge = ?
# After the loop, theta contains the fitted coefficients
theta_gdOLS = ?
theta_gdRidge = ?
print("Gradient Descent OLS coefficients:", theta_gdOLS)
print("Gradient Descent Ridge coefficients:", theta_gdRidge)
!ec
=== 3a) ===
Discuss the results as function of the learning rate parameters and the number of iterations.
=== 3b) ===
Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?
Try to fit the above data set using OLS and Ridge regression with the analytical expressions and your own gradient descent codes.
If everything worked correctly, the learned coefficients should be
close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to