lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-01 08:51:55 +02:00
parent 7bcccbbf71
commit 7bd40a8c65
9 changed files with 589 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb
+59 -42
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "7d56b2d5",
"id": "d3aa801d",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "c7a8e9c7",
"id": "7c64e6da",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "cf8f0ecb",
"id": "51e35698",
"metadata": {
"editable": true
},
@@ -46,7 +46,7 @@
},
{
"cell_type": "markdown",
"id": "a67ae548",
"id": "74fb184e",
"metadata": {
"editable": true
},
@@ -72,7 +72,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "f2d4a55d",
"id": "9e6acfef",
"metadata": {
"collapsed": false,
"editable": true
@@ -101,33 +101,43 @@
},
{
"cell_type": "markdown",
"id": "a445583b",
"id": "f2d03ca8",
"metadata": {
"editable": true
},
"source": [
"This code produces a dataset where only features 0, 1, and 6\n",
"significantly influence $\\boldsymbol{y}$. The rest of the features have zero true\n",
"coefficient, so they only contribute noise. For example, feature 0 has\n",
"coefficient. For example, feature 0 has\n",
"a true weight of 5.0, feature 1 has -3.0, and feature 6 has 2.0, so\n",
"the expected relationship is:"
]
},
{
"cell_type": "markdown",
"id": "4a81ddf9",
"id": "d2d64f9b",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"y \\approx 5 \\times X_0 \\;-\\; 3 \\times X_1 \\;+\\; 2 \\times X_6 \\;+\\; \\text{noise}.\n",
"y \\approx 5 \\times x_0 \\;-\\; 3 \\times x_1 \\;+\\; 2 \\times x_6 \\;+\\; \\text{noise}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "ae590275",
"id": "b4248e9d",
"metadata": {
"editable": true
},
"source": [
"You can remove the noise if you wish to."
]
},
{
"cell_type": "markdown",
"id": "5fed181f",
"metadata": {
"editable": true
},
@@ -138,14 +148,24 @@
"standardize the features. This ensures all features are on a\n",
"comparable scale, which is especially important when using\n",
"regularization. Here we will perform standardization, scaling each\n",
"feature to have mean 0 and standard deviation 1:\n",
"feature to have mean 0 and standard deviation 1."
]
},
{
"cell_type": "markdown",
"id": "6ec0227c",
"metadata": {
"editable": true
},
"source": [
"### 1a)\n",
"\n",
"Compute the mean and standard deviation of each column (feature) in $bm{X}$.\n",
"Compute the mean and standard deviation of each column (feature) in $\\boldsymbol{X}$.\n",
"Subtract the mean and divide by the standard deviation for each feature.\n",
"\n",
"We will also center the target $\\boldsymbol{y}$ to mean $0$. Centering $\\boldsymbol{y}$\n",
"(and each feature) means the model won’t require a separate intercept\n",
"term – the data is shifted such that the intercept is effectively 0\n",
"(and each feature) means the model does not require a separate intercept\n",
"term, the data is shifted such that the intercept is effectively 0\n",
". (In practice, one could include an intercept in the model and not\n",
"penalize it, but here we simplify by centering.)"
]
@@ -153,7 +173,7 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "8b40c47a",
"id": "a140aac7",
"metadata": {
"collapsed": false,
"editable": true
@@ -173,36 +193,34 @@
},
{
"cell_type": "markdown",
"id": "ff9c0c81",
"id": "57ad18f5",
"metadata": {
"editable": true
},
"source": [
"### 1a)\n",
"\n",
"Fill in the necessary details.\n",
"\n",
"After this preprocessing, each column of $\\boldsymbol{X}_norm$ has mean zero and standard deviation $1$\n",
"and $\\boldsymbol{y}_centered$ has mean 0. This makes the optimization landscape\n",
"After this preprocessing, each column of $\\boldsymbol{X}_{\\mathrm{norm}}$ has mean zero and standard deviation $1$\n",
"and $\\boldsymbol{y}_{\\mathrm{centered}}$ has mean 0. This makes the optimization landscape\n",
"nicer and ensures the regularization penalty $\\lambda \\sum_j\n",
"\\beta_j^2$ treats each coefficient fairly (since features are on the\n",
"\\theta_j^2$ in Ridge regression treats each coefficient fairly (since features are on the\n",
"same scale)."
]
},
{
"cell_type": "markdown",
"id": "d27c70e4",
"id": "2886697d",
"metadata": {
"editable": true
},
"source": [
"## Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{theta}$"
"## Exercise 2, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "9f1e5184",
"id": "97ac6cb6",
"metadata": {
"collapsed": false,
"editable": true
@@ -223,34 +241,33 @@
},
{
"cell_type": "markdown",
"id": "2ec556b9",
"id": "3efb067b",
"metadata": {
"editable": true
},
"source": [
"This computes the ridge and OLS regression coefficients directly. The identity\n",
"matrix $I$ has the same size as $X^T X$ (which is n_features x\n",
"n_features), and lam * I adds $\\lambda$ to the diagonal of $X^T X. We\n",
"then invert this matrix and multiply by $X^T y. The result\n",
"for $\\boldsymbol{\\theta}$ is a NumPy array of shape (n_features,) containing the\n",
"fitted weights."
"This computes the Ridge and OLS regression coefficients directly. The identity\n",
"matrix $I$ has the same size as $X^T X$. It adds $\\lambda$ to the diagonal of $X^T X for Ridge regression. We\n",
"then invert this matrix and multiply by $X^T y$. The result\n",
"for $\\boldsymbol{\\theta}$ is a NumPy array of shape (n$\\_$features,) containing the\n",
"fitted parameters $\\boldsymbol{\\theta}$.."
]
},
{
"cell_type": "markdown",
"id": "a821f0c5",
"id": "53be2bf8",
"metadata": {
"editable": true
},
"source": [
"### 2a)\n",
"\n",
"Finalize the OLS and Ridge regression determination of the optimal parameters $bm{\\theta}$."
"Finalize, in the above code, the OLS and Ridge regression determination of the optimal parameters $\\boldsymbol{\\theta}$."
]
},
{
"cell_type": "markdown",
"id": "d637130e",
"id": "e4126591",
"metadata": {
"editable": true
},
@@ -262,12 +279,12 @@
},
{
"cell_type": "markdown",
"id": "b455ce7e",
"id": "642d0850",
"metadata": {
"editable": true
},
"source": [
"## Implementing the simplest form for gradient descent\n",
"## Exercise 3, Implementing the simplest form for gradient descent\n",
"\n",
"Alternatively, we can fit the ridge regression model using gradient\n",
"descent. This is useful to visualize the iterative convergence and is\n",
@@ -282,7 +299,7 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "cfa1eb29",
"id": "a67af634",
"metadata": {
"collapsed": false,
"editable": true
@@ -325,32 +342,32 @@
},
{
"cell_type": "markdown",
"id": "dc78d58d",
"id": "1c8c35dc",
"metadata": {
"editable": true
},
"source": [
"### 3a)\n",
"\n",
"Discuss the results as function of the learning rate paramaters and the number of iterations."
"Discuss the results as function of the learning rate parameters and the number of iterations."
]
},
{
"cell_type": "markdown",
"id": "15060acb",
"id": "899fec5c",
"metadata": {
"editable": true
},
"source": [
"### 3b)\n",
"\n",
"Add a stopping parameter as function of the number iterations. \n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion? \n",
"\n",
"If everything worked correctly, the learned coefficients should be\n",
"close to the true values [5.0, -3.0, 0.0, …, 2.0, …] that we used to\n",
"generate the data. Keep in mind that due to regularization and noise,\n",
"the learned values will not exactly equal the true ones, but they\n",
"should be in the same ballpark."
"should be in the same ballpark. Which method (OLS or Ridge) gives the best results?"
]
}
],