lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

addex exercise week 37

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-01 13:43:58 +02:00
parent dccb74ba38
commit eb31c10d1c
41 changed files with 718 additions and 549 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/_sources/exercisesweek35.ipynb
+1 -1
View File
@@ -323,7 +323,7 @@
"source": [
"n = 100\n",
"x = np.linspace(-3, 3, n)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)"
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 1.0)"
]
},
{
doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb
+146 -94
View File
@@ -2,24 +2,24 @@
"cells": [
{
"cell_type": "markdown",
"id": "d3aa801d",
"id": "b0268cb1",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek37.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 36 -->"
"<!-- dom:TITLE: Exercises week 37 -->"
]
},
{
"cell_type": "markdown",
"id": "7c64e6da",
"id": "700a1d0b",
"metadata": {
"editable": true
},
"source": [
"# Exercises week 36\n",
"# Exercises week 37\n",
"**Implementing gradient descent for Ridge and ordinary Least Squares Regression**\n",
"\n",
"Date: **September 8-12, 2025**"
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "51e35698",
"id": "dbe5809a",
"metadata": {
"editable": true
},
@@ -46,98 +46,44 @@
},
{
"cell_type": "markdown",
"id": "74fb184e",
"id": "ac99e9c0",
"metadata": {
"editable": true
},
"source": [
"## Ridge regression and a new Synthetic Dataset\n",
"## Simple one-dimensional second-order polynomial\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we’ll\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9e6acfef",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
"We start with a very simple function"
]
},
{
"cell_type": "markdown",
"id": "f2d03ca8",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d2d64f9b",
"id": "6d71a32d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"\\f(x)= 2-x+5x^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b4248e9d",
"id": "c6496768",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to."
"defined for $x\\in [-2,2]$. You can add noise if you wish. \n",
"\n",
"We are going to fit this function with a polynomial ansatz. The easiest thing is to set up a second-order polynomial and see if you can fit the above function.\n",
"Feel free to play around with higher-order polynomials."
]
},
{
"cell_type": "markdown",
"id": "5fed181f",
"id": "24678181",
"metadata": {
"editable": true
},
@@ -153,27 +99,28 @@
},
{
"cell_type": "markdown",
"id": "6ec0227c",
"id": "6b1bd90a",
"metadata": {
"editable": true
},
"source": [
"### 1a)\n",
"\n",
"Compute the mean and standard deviation of each column (feature) in $\\boldsymbol{X}$.\n",
"Compute the mean and standard deviation of each column (feature) in your design/feature matrix $\\boldsymbol{X}$.\n",
"Subtract the mean and divide by the standard deviation for each feature.\n",
"\n",
"We will also center the target $\\boldsymbol{y}$ to mean $0$. Centering $\\boldsymbol{y}$\n",
"(and each feature) means the model does not require a separate intercept\n",
"term, the data is shifted such that the intercept is effectively 0\n",
". (In practice, one could include an intercept in the model and not\n",
"penalize it, but here we simplify by centering.)"
"penalize it, but here we simplify by centering.)\n",
"Choose $n=100$ data points and set up $\\boldsymbol{x}, $\\boldsymbol{y} and the design matrix $\\boldsymbol{X}$."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a140aac7",
"execution_count": 1,
"id": "5e751a16",
"metadata": {
"collapsed": false,
"editable": true
@@ -193,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "57ad18f5",
"id": "50a68a52",
"metadata": {
"editable": true
},
@@ -209,18 +156,30 @@
},
{
"cell_type": "markdown",
"id": "2886697d",
"id": "db74a970",
"metadata": {
"editable": true
},
"source": [
"## Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 2, calculate the gradients\n",
"\n",
"Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function."
]
},
{
"cell_type": "markdown",
"id": "f8feaa49",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "97ac6cb6",
"execution_count": 2,
"id": "e6eca7a7",
"metadata": {
"collapsed": false,
"editable": true
@@ -241,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "3efb067b",
"id": "a2dff0cf",
"metadata": {
"editable": true
},
@@ -255,36 +214,36 @@
},
{
"cell_type": "markdown",
"id": "53be2bf8",
"id": "d7542c3e",
"metadata": {
"editable": true
},
"source": [
"### 2a)\n",
"### 3a)\n",
"\n",
"Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters $\\boldsymbol{\\theta}$."
]
},
{
"cell_type": "markdown",
"id": "e4126591",
"id": "b922fd54",
"metadata": {
"editable": true
},
"source": [
"### 2b)\n",
"### 3b)\n",
"\n",
"Explore the results as function of different values of the hyperparameter $\\lambda$. See for example exercise 4 from week 36."
]
},
{
"cell_type": "markdown",
"id": "642d0850",
"id": "92268a9a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, Implementing the simplest form for gradient descent\n",
"## Exercise 4, Implementing the simplest form for gradient descent\n",
"\n",
"Alternatively, we can fit the ridge regression model using gradient\n",
"descent. This is useful to visualize the iterative convergence and is\n",
@@ -298,8 +257,8 @@
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a67af634",
"execution_count": 3,
"id": "81660fa3",
"metadata": {
"collapsed": false,
"editable": true
@@ -342,26 +301,119 @@
},
{
"cell_type": "markdown",
"id": "1c8c35dc",
"id": "95149551",
"metadata": {
"editable": true
},
"source": [
"### 3a)\n",
"### 4a)\n",
"\n",
"Discuss the results as function of the learning rate parameters and the number of iterations."
]
},
{
"cell_type": "markdown",
"id": "899fec5c",
"id": "09b5400e",
"metadata": {
"editable": true
},
"source": [
"### 3b)\n",
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion? \n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "c635ca9e",
"metadata": {
"editable": true
},
"source": [
"## Exercise 5, Ridge regression and a new Synthetic Dataset\n",
"\n",
"We create a synthetic linear regression dataset with a sparse\n",
"underlying relationship. This means we have many features but only a\n",
"few of them actually contribute to the target. In our example, we’ll\n",
"use 10 features with only 3 non-zero weights in the true model. This\n",
"way, the target is generated as a linear combination of a few features\n",
"(with known coefficients) plus some random noise. The steps we include are:\n",
"\n",
"Decide on the number of samples and features (e.g. 100 samples, 10 features).\n",
"Define the **true** coefficient vector with mostly zeros (for sparsity). For example, we set $\\hat{\\boldsymbol{\\theta}} = [5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0]$, meaning only features 0, 1, and 6 have a real effect on y.\n",
"\n",
"Then we sample feature values for $\\boldsymbol{X}$ randomly (e.g. from a normal distribution). We use a normal distribution so features are roughly centered around 0.\n",
"Then we compute the target values $y$ using the linear combination $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and add some noise (to simulate measurement error or unexplained variance).\n",
"\n",
"Below is the code to generate the dataset:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4ca4ce05",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# Set random seed for reproducibility\n",
"np.random.seed(0)\n",
"\n",
"# Define dataset size\n",
"n_samples = 100\n",
"n_features = 10\n",
"\n",
"# Define true coefficients (sparse linear relationship)\n",
"theta_true = np.array([5.0, -3.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0])\n",
"\n",
"# Generate feature matrix X (n_samples x n_features) with random values\n",
"X = np.random.randn(n_samples, n_features) # standard normal distribution\n",
"\n",
"# Generate target values y with a linear combination of X and theta_true, plus noise\n",
"noise = 0.5 * np.random.randn(n_samples) # Gaussian noise\n",
"y = X.dot @ theta_true + noise"
]
},
{
"cell_type": "markdown",
"id": "af39d8bc",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "d35d3438",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b28bf122",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to. \n",
"\n",
"Try to fit the above data set using OLS and Ridge regression with the analytical expressions and your own gradient descent codes.\n",
"\n",
"If everything worked correctly, the learned coefficients should be\n",
"close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to\n",