From 59c413e03b8fd45c8c572ffb433bdaed9f949a15 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 19 Sep 2022 07:48:41 +0200 Subject: [PATCH] Update week36.ipynb --- doc/pub/week36/ipynb/week36.ipynb | 1347 ++++++++--------------------- 1 file changed, 379 insertions(+), 968 deletions(-) diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 4f5ffb301..efa50c1ae 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "54f68878", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "b2606fb6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 36: Statistical interpretation of Linear Regression and Resampling techniques\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "998031f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plans for week 36\n", "\n", @@ -55,9 +49,7 @@ { "cell_type": "markdown", "id": "6cd06197", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Thursday September 8" ] @@ -65,9 +57,7 @@ { "cell_type": "markdown", "id": "d54cdcb7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples" ] @@ -75,9 +65,7 @@ { "cell_type": "markdown", "id": "e00e68d6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linear Regression and the SVD\n", "\n", @@ -87,9 +75,7 @@ { "cell_type": "markdown", "id": "34b87b68", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -99,9 +85,7 @@ { "cell_type": "markdown", "id": "ce084898", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined last week the matrix" ] @@ -109,9 +93,7 @@ { "cell_type": "markdown", "id": "2976c1d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\end{bmatrix},\n", @@ -121,9 +103,7 @@ { "cell_type": "markdown", "id": "6a83cdb5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] @@ -131,9 +111,7 @@ { "cell_type": "markdown", "id": "a7a4fa16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -148,9 +126,7 @@ { "cell_type": "markdown", "id": "03e6fafb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning we can write" ] @@ -158,9 +134,7 @@ { "cell_type": "markdown", "id": "7b638f4f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -170,9 +144,7 @@ { "cell_type": "markdown", "id": "ec26d490", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] @@ -180,9 +152,7 @@ { "cell_type": "markdown", "id": "89a41f1b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -192,9 +162,7 @@ { "cell_type": "markdown", "id": "1a1ec91f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What does it mean?\n", "\n", @@ -206,9 +174,7 @@ { "cell_type": "markdown", "id": "9be07751", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -218,9 +184,7 @@ { "cell_type": "markdown", "id": "71861426", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -240,9 +204,7 @@ { "cell_type": "markdown", "id": "8696c3c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -252,9 +214,7 @@ { "cell_type": "markdown", "id": "9cf87ecc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", @@ -267,9 +227,7 @@ { "cell_type": "markdown", "id": "20032abb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", @@ -279,9 +237,7 @@ { "cell_type": "markdown", "id": "fbcc9b3f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -291,9 +247,7 @@ { "cell_type": "markdown", "id": "1e18e19f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] @@ -301,9 +255,7 @@ { "cell_type": "markdown", "id": "747b9d45", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -313,9 +265,7 @@ { "cell_type": "markdown", "id": "134925f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "leading to" ] @@ -323,9 +273,7 @@ { "cell_type": "markdown", "id": "0050225b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -335,9 +283,7 @@ { "cell_type": "markdown", "id": "8ec325de", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] @@ -345,9 +291,7 @@ { "cell_type": "markdown", "id": "5059b729", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -357,9 +301,7 @@ { "cell_type": "markdown", "id": "1815abe6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -374,9 +316,7 @@ { "cell_type": "markdown", "id": "ade9b7ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code for SVD and Inversion of Matrices\n", "\n", @@ -388,10 +328,7 @@ "cell_type": "code", "execution_count": 1, "id": "24749dad", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "Ainv = np.linlag.pinv(A)" @@ -400,22 +337,39 @@ { "cell_type": "markdown", "id": "770bef7d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD." ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 4, "id": "23a63ba1", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[ 1. -1. 2.]\n", + " [ 1. 0. 1.]\n", + " [ 1. 2. -1.]\n", + " [ 1. 1. 0.]]\n", + "test U\n", + "[[-3.33066907e-16 -1.11022302e-16 3.33066907e-16]\n", + " [-1.11022302e-16 4.44089210e-16 -2.49800181e-16]\n", + " [ 3.33066907e-16 -2.49800181e-16 0.00000000e+00]]\n", + "test VT\n", + "[[ 2.22044605e-16 5.55111512e-17 -2.22044605e-16]\n", + " [ 5.55111512e-17 -2.22044605e-16 5.55111512e-17]\n", + " [-2.22044605e-16 5.55111512e-17 2.22044605e-16]]\n", + "[[1.82969604e+30 1.82969604e+30 1.82969604e+30]\n", + " [1.82969604e+30 1.82969604e+30 1.82969604e+30]\n", + " [1.82969604e+30 1.82969604e+30 1.82969604e+30]]\n" + ] + } + ], "source": [ "import numpy as np\n", "# SVD inversion\n", @@ -437,13 +391,13 @@ " return np.matmul(V,np.matmul(invD,UT))\n", "\n", "\n", - "#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n", + "X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n", "# Non-singular square matrix\n", - "X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n", + "#X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n", "print(X)\n", "A = np.transpose(X) @ X\n", "# Brute force inversion\n", - "B = np.linalg.inv(A) # here we could use np.linalg.pinv(A)\n", + "B = np.linalg.pinv(A) # here we could use np.linalg.pinv(A)\n", "C = SVDinv(A)\n", "print(np.abs(B-C))" ] @@ -451,9 +405,7 @@ { "cell_type": "markdown", "id": "7bd93f48", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Inverse of Rectangular Matrix\n", "\n", @@ -471,9 +423,7 @@ { "cell_type": "markdown", "id": "3fd0e27b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}_{\\mathrm{PI}}= \\boldsymbol{V}\\boldsymbol{D}_{\\mathrm{PI}}\\boldsymbol{U}^T,\n", @@ -483,22 +433,32 @@ { "cell_type": "markdown", "id": "067d781b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{D}_{\\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\\boldsymbol{\\Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD." ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "id": "c17f9502", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[0.3 0.4]\n", + " [0.5 0.6]\n", + " [0.7 0.8]\n", + " [0.9 1. ]]\n", + "[[-13. -6. 1. 8. ]\n", + " [ 11.5 5.5 -0.5 -6.5]]\n", + "[[0. 0. 0. 0.]\n", + " [0. 0. 0. 0.]]\n" + ] + } + ], "source": [ "import numpy as np\n", "# SVD inversion\n", @@ -528,9 +488,7 @@ { "cell_type": "markdown", "id": "c9e2f968", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by **Numpy**." ] @@ -538,9 +496,7 @@ { "cell_type": "markdown", "id": "715bfc44", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and LASSO Regression\n", "\n", @@ -551,9 +507,7 @@ { "cell_type": "markdown", "id": "a7250920", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -563,9 +517,7 @@ { "cell_type": "markdown", "id": "2d4eca08", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or we can state it as" ] @@ -573,9 +525,7 @@ { "cell_type": "markdown", "id": "5c47d606", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -586,9 +536,7 @@ { "cell_type": "markdown", "id": "be48173a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used the definition of a norm-2 vector, that is" ] @@ -596,9 +544,7 @@ { "cell_type": "markdown", "id": "c1cc0c52", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -608,9 +554,7 @@ { "cell_type": "markdown", "id": "85043b85", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## From OLS to Ridge and Lasso\n", "\n", @@ -623,9 +567,7 @@ { "cell_type": "markdown", "id": "8b28ed2a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -636,9 +578,7 @@ { "cell_type": "markdown", "id": "498c8111", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -648,9 +588,7 @@ { "cell_type": "markdown", "id": "9d41fbd8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -660,9 +598,7 @@ { "cell_type": "markdown", "id": "6bacb19c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have a new optimization equation" ] @@ -670,9 +606,7 @@ { "cell_type": "markdown", "id": "4e386cd1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -683,9 +617,7 @@ { "cell_type": "markdown", "id": "1bd20824", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -695,9 +627,7 @@ { "cell_type": "markdown", "id": "4264b9bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -707,9 +637,7 @@ { "cell_type": "markdown", "id": "eec0c847", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -719,9 +647,7 @@ { "cell_type": "markdown", "id": "e11a479f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -731,9 +657,7 @@ { "cell_type": "markdown", "id": "0f614521", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -745,9 +669,7 @@ { "cell_type": "markdown", "id": "e0a2c65a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -757,9 +679,7 @@ { "cell_type": "markdown", "id": "9a36d518", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] @@ -767,9 +687,7 @@ { "cell_type": "markdown", "id": "ec99d6ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -779,9 +697,7 @@ { "cell_type": "markdown", "id": "e5102c81", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $t$ a finite positive number." ] @@ -789,9 +705,7 @@ { "cell_type": "markdown", "id": "277c29d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Note on Scikit-Learn\n", "\n", @@ -801,9 +715,7 @@ { "cell_type": "markdown", "id": "3ee1172e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -813,9 +725,7 @@ { "cell_type": "markdown", "id": "5555becd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In our codes where we compare our own codes with **Scikit-Learn**, we do thus not include the $1/n$ factor in the cost function." ] @@ -823,9 +733,7 @@ { "cell_type": "markdown", "id": "bf920b7a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Comparison with OLS\n", "When we compare this with the ordinary least squares result we have" @@ -834,9 +742,7 @@ { "cell_type": "markdown", "id": "dcc8b3a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -846,9 +752,7 @@ { "cell_type": "markdown", "id": "b51d29a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", @@ -862,9 +766,7 @@ { "cell_type": "markdown", "id": "4c81e874", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## SVD analysis\n", "\n", @@ -875,9 +777,7 @@ { "cell_type": "markdown", "id": "7e9a88c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -887,9 +787,7 @@ { "cell_type": "markdown", "id": "16c44a1f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For Ridge regression this becomes" ] @@ -897,9 +795,7 @@ { "cell_type": "markdown", "id": "be091747", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -909,9 +805,7 @@ { "cell_type": "markdown", "id": "e7c4958c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." ] @@ -919,9 +813,7 @@ { "cell_type": "markdown", "id": "096042d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpreting the Ridge results\n", "\n", @@ -931,9 +823,7 @@ { "cell_type": "markdown", "id": "a6133162", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -943,9 +833,7 @@ { "cell_type": "markdown", "id": "de47ccce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -959,9 +847,7 @@ { "cell_type": "markdown", "id": "38d8e676", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More interpretations\n", "\n", @@ -971,9 +857,7 @@ { "cell_type": "markdown", "id": "cf226622", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -983,9 +867,7 @@ { "cell_type": "markdown", "id": "e51ab72f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In this case the standard OLS results in" ] @@ -993,9 +875,7 @@ { "cell_type": "markdown", "id": "aded2c2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", @@ -1005,9 +885,7 @@ { "cell_type": "markdown", "id": "d18586ec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1015,9 +893,7 @@ { "cell_type": "markdown", "id": "d60836ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -1027,9 +903,7 @@ { "cell_type": "markdown", "id": "83ee48f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -1044,9 +918,7 @@ { "cell_type": "markdown", "id": "6379bc41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving the Lasso Regression Equations\n", "\n", @@ -1056,9 +928,7 @@ { "cell_type": "markdown", "id": "60a42dcc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -1068,9 +938,7 @@ { "cell_type": "markdown", "id": "d335d8fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] @@ -1078,9 +946,7 @@ { "cell_type": "markdown", "id": "f7988d24", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -1090,9 +956,7 @@ { "cell_type": "markdown", "id": "455c0597", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have that the derivative of the cost function is" ] @@ -1100,9 +964,7 @@ { "cell_type": "markdown", "id": "31b80ae8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -1112,9 +974,7 @@ { "cell_type": "markdown", "id": "89cfcd2e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and reordering we have" ] @@ -1122,9 +982,7 @@ { "cell_type": "markdown", "id": "7b9c1855", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1134,9 +992,7 @@ { "cell_type": "markdown", "id": "64b3eb85", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." ] @@ -1144,9 +1000,7 @@ { "cell_type": "markdown", "id": "57e33e83", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression\n", "\n", @@ -1159,9 +1013,7 @@ { "cell_type": "markdown", "id": "9aaf5adc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n", @@ -1171,9 +1023,7 @@ { "cell_type": "markdown", "id": "a7e0855d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -1181,9 +1031,7 @@ { "cell_type": "markdown", "id": "282ec43f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n", @@ -1193,9 +1041,7 @@ { "cell_type": "markdown", "id": "9a8686a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge Regression\n", "\n", @@ -1205,9 +1051,7 @@ { "cell_type": "markdown", "id": "d0c7441a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", @@ -1217,9 +1061,7 @@ { "cell_type": "markdown", "id": "123a953a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -1227,9 +1069,7 @@ { "cell_type": "markdown", "id": "b80e95f5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n", @@ -1239,9 +1079,7 @@ { "cell_type": "markdown", "id": "1272391e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso Regression\n", "\n", @@ -1251,9 +1089,7 @@ { "cell_type": "markdown", "id": "8a65a800", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", @@ -1263,9 +1099,7 @@ { "cell_type": "markdown", "id": "129b29dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -1273,9 +1107,7 @@ { "cell_type": "markdown", "id": "9671fc3b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", @@ -1285,9 +1117,7 @@ { "cell_type": "markdown", "id": "645d20a1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to" ] @@ -1295,9 +1125,7 @@ { "cell_type": "markdown", "id": "4a940735", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", @@ -1309,9 +1137,7 @@ { "cell_type": "markdown", "id": "f5d24fed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$." ] @@ -1319,9 +1145,7 @@ { "cell_type": "markdown", "id": "56fe26cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Yet another Example\n", "\n", @@ -1331,9 +1155,7 @@ { "cell_type": "markdown", "id": "bcdecc9d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}=\\begin{bmatrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n", @@ -1343,9 +1165,7 @@ { "cell_type": "markdown", "id": "088b5e9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and our inputs as a $3\\times 2$ design matrix" ] @@ -1353,9 +1173,7 @@ { "cell_type": "markdown", "id": "813decf7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n", @@ -1365,9 +1183,7 @@ { "cell_type": "markdown", "id": "d2887852", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression." ] @@ -1375,9 +1191,7 @@ { "cell_type": "markdown", "id": "9d7b7953", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The OLS case\n", "\n", @@ -1387,9 +1201,7 @@ { "cell_type": "markdown", "id": "e837f852", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1399,9 +1211,7 @@ { "cell_type": "markdown", "id": "15d802eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Inserting the above values we obtain that" ] @@ -1409,9 +1219,7 @@ { "cell_type": "markdown", "id": "cd4b96b1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", @@ -1421,9 +1229,7 @@ { "cell_type": "markdown", "id": "434b97ea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The code which implements this simpler case is presented after the discussion of Ridge and Lasso." ] @@ -1431,9 +1237,7 @@ { "cell_type": "markdown", "id": "d640a5fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Ridge case\n", "\n", @@ -1443,9 +1247,7 @@ { "cell_type": "markdown", "id": "f8a077e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1455,9 +1257,7 @@ { "cell_type": "markdown", "id": "30897a80", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Inserting the above values we obtain that" ] @@ -1465,9 +1265,7 @@ { "cell_type": "markdown", "id": "1e751907", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", @@ -1477,9 +1275,7 @@ { "cell_type": "markdown", "id": "ca66337b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n", @@ -1490,9 +1286,7 @@ { "cell_type": "markdown", "id": "30cb3a5b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Writing the Cost Function\n", "\n", @@ -1502,9 +1296,7 @@ { "cell_type": "markdown", "id": "365400dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n", @@ -1514,9 +1306,7 @@ { "cell_type": "markdown", "id": "f09ebaf1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n", @@ -1526,9 +1316,7 @@ { "cell_type": "markdown", "id": "1de97d2c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and taking the derivative with respect to $\\beta_0$ we get" ] @@ -1536,9 +1324,7 @@ { "cell_type": "markdown", "id": "78632ac0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0=\\frac{8}{4+\\lambda},\n", @@ -1548,9 +1334,7 @@ { "cell_type": "markdown", "id": "3c88d85f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and for $\\beta_1$ we obtain" ] @@ -1558,9 +1342,7 @@ { "cell_type": "markdown", "id": "97140392", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_1=\\frac{2}{1+\\lambda},\n", @@ -1570,9 +1352,7 @@ { "cell_type": "markdown", "id": "7d1ea782", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving" ] @@ -1580,9 +1360,7 @@ { "cell_type": "markdown", "id": "9b49be00", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n", @@ -1592,9 +1370,7 @@ { "cell_type": "markdown", "id": "7bb324bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$." ] @@ -1602,9 +1378,7 @@ { "cell_type": "markdown", "id": "610a2a56", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso case\n", "\n", @@ -1615,9 +1389,7 @@ { "cell_type": "markdown", "id": "fc93bd4d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n", @@ -1627,9 +1399,7 @@ { "cell_type": "markdown", "id": "7ac09787", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n", @@ -1639,9 +1409,7 @@ { "cell_type": "markdown", "id": "a7b2f817", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1649,9 +1417,7 @@ { "cell_type": "markdown", "id": "99632928", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n", @@ -1661,9 +1427,7 @@ { "cell_type": "markdown", "id": "94c7a534", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n", "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n", @@ -1678,9 +1442,7 @@ { "cell_type": "markdown", "id": "e55fc2b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The first Case\n", "\n", @@ -1690,9 +1452,7 @@ { "cell_type": "markdown", "id": "e2ef7286", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-4(4-2\\beta_0)+\\lambda=0,\n", @@ -1702,9 +1462,7 @@ { "cell_type": "markdown", "id": "3a5c871e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1712,9 +1470,7 @@ { "cell_type": "markdown", "id": "b9ca6fd3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-2(2-\\beta_1)+\\lambda=0.\n", @@ -1724,9 +1480,7 @@ { "cell_type": "markdown", "id": "4c69db28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which yields" ] @@ -1734,9 +1488,7 @@ { "cell_type": "markdown", "id": "c77c93cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0=\\frac{16+\\lambda}{8},\n", @@ -1746,9 +1498,7 @@ { "cell_type": "markdown", "id": "cfdc629c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1756,9 +1506,7 @@ { "cell_type": "markdown", "id": "07e71e01", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_1=\\frac{4+\\lambda}{2}.\n", @@ -1768,9 +1516,7 @@ { "cell_type": "markdown", "id": "c8f65682", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you." ] @@ -1778,9 +1524,7 @@ { "cell_type": "markdown", "id": "bc6233b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple code for solving the above problem\n", "\n", @@ -1793,10 +1537,7 @@ "cell_type": "code", "execution_count": 4, "id": "d594a12f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -1855,9 +1596,7 @@ { "cell_type": "markdown", "id": "6bd62e0a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see here that we reach a plateau. What is actually happening?" ] @@ -1865,9 +1604,7 @@ { "cell_type": "markdown", "id": "715aeaa5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## With Lasso Regression" ] @@ -1876,10 +1613,7 @@ "cell_type": "code", "execution_count": 5, "id": "84affb9e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import os\n", @@ -1943,9 +1677,7 @@ { "cell_type": "markdown", "id": "70698b7b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Another Example, now with a polynomial fit" ] @@ -1954,10 +1686,7 @@ "cell_type": "code", "execution_count": 6, "id": "d5e411b4", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import os\n", @@ -2043,9 +1772,7 @@ { "cell_type": "markdown", "id": "10b0a7eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## To think about, first part\n", "\n", @@ -2071,9 +1798,7 @@ { "cell_type": "markdown", "id": "0fdacb5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More thinking\n", "\n", @@ -2106,9 +1831,7 @@ { "cell_type": "markdown", "id": "6ade509a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Still thinking\n", "\n", @@ -2120,10 +1843,7 @@ "cell_type": "code", "execution_count": 7, "id": "ca8d02c2", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "#Model training, we compute the mean value of y and X\n", @@ -2145,9 +1865,7 @@ { "cell_type": "markdown", "id": "bb52b09b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What does centering (subtracting the mean values) mean mathematically?\n", "\n", @@ -2161,9 +1879,7 @@ { "cell_type": "markdown", "id": "3cde4c58", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", @@ -2173,9 +1889,7 @@ { "cell_type": "markdown", "id": "e0e6cac8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.\n", "\n", @@ -2188,9 +1902,7 @@ { "cell_type": "markdown", "id": "1e631321", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", @@ -2200,9 +1912,7 @@ { "cell_type": "markdown", "id": "8e5038d9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "for all $j$. For $\\beta_0$ we have" ] @@ -2210,9 +1920,7 @@ { "cell_type": "markdown", "id": "bb56c7ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", @@ -2222,9 +1930,7 @@ { "cell_type": "markdown", "id": "3ed4442a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying away the constant $2/n$, we obtain" ] @@ -2232,9 +1938,7 @@ { "cell_type": "markdown", "id": "9104f8b7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", @@ -2244,9 +1948,7 @@ { "cell_type": "markdown", "id": "4a12287c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Further Manipulations\n", "\n", @@ -2257,9 +1959,7 @@ { "cell_type": "markdown", "id": "5b425023", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", @@ -2269,9 +1969,7 @@ { "cell_type": "markdown", "id": "e6fa30b0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We obtain then" ] @@ -2279,9 +1977,7 @@ { "cell_type": "markdown", "id": "28be70e6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", @@ -2291,9 +1987,7 @@ { "cell_type": "markdown", "id": "2b41476d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we define" ] @@ -2301,9 +1995,7 @@ { "cell_type": "markdown", "id": "97b05bfa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mu_1=\\frac{1}{n}\\sum_{i=0}^{n-1} (X_{i1},\n", @@ -2313,9 +2005,7 @@ { "cell_type": "markdown", "id": "bbb7f94a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and if we define the mean value of the outputs as" ] @@ -2323,9 +2013,7 @@ { "cell_type": "markdown", "id": "c0d06b59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n", @@ -2335,9 +2023,7 @@ { "cell_type": "markdown", "id": "da1e1b48", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have" ] @@ -2345,9 +2031,7 @@ { "cell_type": "markdown", "id": "b840be0b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\mu_y - \\beta_1\\mu_{1}.\n", @@ -2357,9 +2041,7 @@ { "cell_type": "markdown", "id": "c034bf6b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the general case, that is we have more parameters than $\\beta_0$ and $\\beta_1$, we have" ] @@ -2367,9 +2049,7 @@ { "cell_type": "markdown", "id": "8f64ec46", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", @@ -2379,9 +2059,7 @@ { "cell_type": "markdown", "id": "dea8f0e3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)" ] @@ -2389,9 +2067,7 @@ { "cell_type": "markdown", "id": "7d83c4bd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", @@ -2401,9 +2077,7 @@ { "cell_type": "markdown", "id": "bff682c6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Wrapping it up\n", "\n", @@ -2413,9 +2087,7 @@ { "cell_type": "markdown", "id": "be879f0d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", @@ -2425,9 +2097,7 @@ { "cell_type": "markdown", "id": "c6f4c9e0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", @@ -2438,9 +2108,7 @@ { "cell_type": "markdown", "id": "d82a7342", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", @@ -2450,9 +2118,7 @@ { "cell_type": "markdown", "id": "2917e850", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "What does this mean? And why do we insist on all this? Let us look at some examples." ] @@ -2460,9 +2126,7 @@ { "cell_type": "markdown", "id": "50b08ee1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linear Regression code, Intercept handling first\n", "\n", @@ -2474,10 +2138,7 @@ "cell_type": "code", "execution_count": 8, "id": "2c599d3e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -2571,9 +2232,7 @@ { "cell_type": "markdown", "id": "b27a10b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The intercept is the value of our output/target variable\n", "when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). \n", @@ -2592,9 +2251,7 @@ { "cell_type": "markdown", "id": "b217703e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", @@ -2604,9 +2261,7 @@ { "cell_type": "markdown", "id": "04f6b527", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "but when we take out the intercept, this equation becomes" ] @@ -2614,9 +2269,7 @@ { "cell_type": "markdown", "id": "ca6755df", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", @@ -2626,9 +2279,7 @@ { "cell_type": "markdown", "id": "3e2e804d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For Lasso regression we have" ] @@ -2636,9 +2287,7 @@ { "cell_type": "markdown", "id": "b28a22e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", @@ -2648,9 +2297,7 @@ { "cell_type": "markdown", "id": "b580959b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get a MSE which then contains the intercept." ] @@ -2658,9 +2305,7 @@ { "cell_type": "markdown", "id": "0a7136a9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code Examples\n", "\n", @@ -2671,10 +2316,7 @@ "cell_type": "code", "execution_count": 9, "id": "fb95e6e4", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -2747,9 +2389,7 @@ { "cell_type": "markdown", "id": "345ca257", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n", "We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n", @@ -2760,9 +2400,7 @@ { "cell_type": "markdown", "id": "0c591f94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Taking out the mean" ] @@ -2771,10 +2409,7 @@ "cell_type": "code", "execution_count": 10, "id": "d9dbc988", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -2860,9 +2495,7 @@ { "cell_type": "markdown", "id": "9cd41fb5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see here, when compared to the code which includes explicitely the\n", "intercept column, that our MSE value is actually smaller. This is\n", @@ -2877,9 +2510,7 @@ { "cell_type": "markdown", "id": "85a6e4f5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Friday September 9" ] @@ -2887,9 +2518,7 @@ { "cell_type": "markdown", "id": "a2a65ad6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linking the regression analysis with a statistical interpretation\n", "\n", @@ -2916,9 +2545,7 @@ { "cell_type": "markdown", "id": "e8d65034", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -2932,9 +2559,7 @@ { "cell_type": "markdown", "id": "6fa8ce61", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The randomness of $\\varepsilon_i$ implies that\n", "$\\mathbf{y}_i$ is also a random variable. In particular,\n", @@ -2951,9 +2576,7 @@ { "cell_type": "markdown", "id": "e503e716", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Assumptions made\n", "\n", @@ -2965,9 +2588,7 @@ { "cell_type": "markdown", "id": "6590f6c6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -2977,9 +2598,7 @@ { "cell_type": "markdown", "id": "36c66ae3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We approximate this function with our model from the solution of the linear regression equations, that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with" @@ -2988,9 +2607,7 @@ { "cell_type": "markdown", "id": "c794166a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -3000,9 +2617,7 @@ { "cell_type": "markdown", "id": "23801ca2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Expectation value and variance\n", "\n", @@ -3012,9 +2627,7 @@ { "cell_type": "markdown", "id": "859a80be", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -3028,9 +2641,7 @@ { "cell_type": "markdown", "id": "8df27f31", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "while\n", "its variance is" @@ -3039,9 +2650,7 @@ { "cell_type": "markdown", "id": "efab0a0d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", @@ -3062,9 +2671,7 @@ { "cell_type": "markdown", "id": "c41b6e91", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)." @@ -3073,9 +2680,7 @@ { "cell_type": "markdown", "id": "2f7a7871", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n", "\n", @@ -3085,9 +2690,7 @@ { "cell_type": "markdown", "id": "0008f742", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", @@ -3097,9 +2700,7 @@ { "cell_type": "markdown", "id": "72111c95", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This means that the estimator of the regression parameters is unbiased.\n", "\n", @@ -3111,9 +2712,7 @@ { "cell_type": "markdown", "id": "52790c16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -3142,9 +2741,7 @@ { "cell_type": "markdown", "id": "864688c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", @@ -3164,9 +2761,7 @@ { "cell_type": "markdown", "id": "37738b35", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", @@ -3176,9 +2771,7 @@ { "cell_type": "markdown", "id": "aeff32a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see clearly that \n", "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", @@ -3189,9 +2782,7 @@ { "cell_type": "markdown", "id": "3321cbc4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", @@ -3201,9 +2792,7 @@ { "cell_type": "markdown", "id": "486213d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", "\n", @@ -3213,9 +2802,7 @@ { "cell_type": "markdown", "id": "ca94083c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", @@ -3225,9 +2812,7 @@ { "cell_type": "markdown", "id": "167df200", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The difference is non-negative definite since each component of the\n", "matrix product is non-negative definite. \n", @@ -3237,9 +2822,7 @@ { "cell_type": "markdown", "id": "b8c4a1e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving OLS from a probability distribution\n", "\n", @@ -3260,9 +2843,7 @@ { "cell_type": "markdown", "id": "148b3cdb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3272,9 +2853,7 @@ { "cell_type": "markdown", "id": "686d7fa3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Independent and Identically Distrubuted (iid)\n", "\n", @@ -3285,9 +2864,7 @@ { "cell_type": "markdown", "id": "17caa70f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n", @@ -3297,9 +2874,7 @@ { "cell_type": "markdown", "id": "011cc0f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n", "\n", @@ -3309,9 +2884,7 @@ { "cell_type": "markdown", "id": "fd47caec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n", @@ -3321,9 +2894,7 @@ { "cell_type": "markdown", "id": "be92a8b7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We will write this in a more compact form reserving $\\boldsymbol{D}$ for the domain of events, including the ouputs (targets) and the inputs. That is\n", "in case we have a simple one-dimensional input and output case" @@ -3332,9 +2903,7 @@ { "cell_type": "markdown", "id": "7af51d2b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})].\n", @@ -3344,9 +2913,7 @@ { "cell_type": "markdown", "id": "856dff7d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the more general case the various inputs should be replaced by the possible features represented by the input data set $\\boldsymbol{X}$. \n", "We can now rewrite the above probability as" @@ -3355,9 +2922,7 @@ { "cell_type": "markdown", "id": "5a446379", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3367,9 +2932,7 @@ { "cell_type": "markdown", "id": "4204c439", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$." ] @@ -3377,9 +2940,7 @@ { "cell_type": "markdown", "id": "d3bc6574", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Maximum Likelihood Estimation (MLE)\n", "\n", @@ -3408,9 +2969,7 @@ { "cell_type": "markdown", "id": "5605b443", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A new Cost Function\n", "\n", @@ -3420,9 +2979,7 @@ { "cell_type": "markdown", "id": "7eb23f48", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n", @@ -3432,9 +2989,7 @@ { "cell_type": "markdown", "id": "6167023f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which becomes" ] @@ -3442,9 +2997,7 @@ { "cell_type": "markdown", "id": "a4443fe7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n", @@ -3454,9 +3007,7 @@ { "cell_type": "markdown", "id": "358b1b20", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely" ] @@ -3464,9 +3015,7 @@ { "cell_type": "markdown", "id": "50135271", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n", @@ -3476,9 +3025,7 @@ { "cell_type": "markdown", "id": "4f04c33f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the well-known OLS equation for the optimal paramters $\\beta$" ] @@ -3486,9 +3033,7 @@ { "cell_type": "markdown", "id": "9adef013", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", @@ -3498,9 +3043,7 @@ { "cell_type": "markdown", "id": "88521aab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics." ] @@ -3508,9 +3051,7 @@ { "cell_type": "markdown", "id": "48e28fb9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More basic Statistics and Bayes' theorem\n", "\n", @@ -3528,9 +3069,7 @@ { "cell_type": "markdown", "id": "d75b9c96", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n", @@ -3540,9 +3079,7 @@ { "cell_type": "markdown", "id": "597adf34", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**The product rule (aka joint probability) is given by.**" ] @@ -3550,9 +3087,7 @@ { "cell_type": "markdown", "id": "ae73334c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X \\cup Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n", @@ -3562,9 +3097,7 @@ { "cell_type": "markdown", "id": "6b91371c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n", "\n", @@ -3574,9 +3107,7 @@ { "cell_type": "markdown", "id": "60c017df", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Marginal Probability\n", "\n", @@ -3586,9 +3117,7 @@ { "cell_type": "markdown", "id": "bfc5883c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X)=\\sum_{i=0}^{n-1}p(X,Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert Y=y_i)p(Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert y_i)p(y_i).\n", @@ -3598,9 +3127,7 @@ { "cell_type": "markdown", "id": "c5227ac5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditional Probability\n", "\n", @@ -3610,9 +3137,7 @@ { "cell_type": "markdown", "id": "6e13d49f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)}=\\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}.\n", @@ -3622,9 +3147,7 @@ { "cell_type": "markdown", "id": "9f576b4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bayes' Theorem\n", "\n", @@ -3634,9 +3157,7 @@ { "cell_type": "markdown", "id": "79f8fe23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)},\n", @@ -3646,9 +3167,7 @@ { "cell_type": "markdown", "id": "b5b3d792", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we can rewrite as" ] @@ -3656,9 +3175,7 @@ { "cell_type": "markdown", "id": "1fff2eb5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}=\\frac{p(Y\\vert X)p(X)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)},\n", @@ -3668,9 +3185,7 @@ { "cell_type": "markdown", "id": "ebfbf1a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$." ] @@ -3678,9 +3193,7 @@ { "cell_type": "markdown", "id": "99f96e4d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpretations of Bayes' Theorem\n", "\n", @@ -3697,9 +3210,7 @@ { "cell_type": "markdown", "id": "46017fbc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Example of Usage of Bayes' theorem\n", "\n", @@ -3718,9 +3229,7 @@ { "cell_type": "markdown", "id": "45acf8d6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X=1\\vert Y=1) =0.8.\n", @@ -3730,9 +3239,7 @@ { "cell_type": "markdown", "id": "d7189c6c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of $80\\%$ for having cancer.\n", "It is however not correct, as the following Bayesian analysis shows." @@ -3741,9 +3248,7 @@ { "cell_type": "markdown", "id": "095f1344", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Doing it correctly\n", "\n", @@ -3754,9 +3259,7 @@ { "cell_type": "markdown", "id": "7a421514", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(Y=1) =0.004.\n", @@ -3766,9 +3269,7 @@ { "cell_type": "markdown", "id": "f6cb9991", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have" ] @@ -3776,9 +3277,7 @@ { "cell_type": "markdown", "id": "4ca65a78", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X=1\\vert Y=0) =0.1.\n", @@ -3788,9 +3287,7 @@ { "cell_type": "markdown", "id": "940490f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute" ] @@ -3798,9 +3295,7 @@ { "cell_type": "markdown", "id": "fd376a0c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}=\\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n", @@ -3810,9 +3305,7 @@ { "cell_type": "markdown", "id": "891b9eda", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "That is, in case of a positive test, there is only a $3\\%$ chance of having breast cancer!" ] @@ -3820,9 +3313,7 @@ { "cell_type": "markdown", "id": "3734a062", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bayes' Theorem and Ridge and Lasso Regression\n", "\n", @@ -3837,9 +3328,7 @@ { "cell_type": "markdown", "id": "2b6262c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Test Function for what happens with OLS, Ridge and Lasso\n", "\n", @@ -3856,10 +3345,7 @@ "cell_type": "code", "execution_count": 11, "id": "6028f12f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -3930,9 +3416,7 @@ { "cell_type": "markdown", "id": "1e94a8e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "How can we understand this?" ] @@ -3940,9 +3424,7 @@ { "cell_type": "markdown", "id": "e89f9a23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Invoking Bayes' theorem\n", "\n", @@ -3954,9 +3436,7 @@ { "cell_type": "markdown", "id": "bcc23e22", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})],\n", @@ -3966,9 +3446,7 @@ { "cell_type": "markdown", "id": "ab35b334", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "is given by" ] @@ -3976,9 +3454,7 @@ { "cell_type": "markdown", "id": "7bad9dac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3988,9 +3464,7 @@ { "cell_type": "markdown", "id": "d58ce44e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" ] @@ -3998,9 +3472,7 @@ { "cell_type": "markdown", "id": "e302a211", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n", @@ -4010,9 +3482,7 @@ { "cell_type": "markdown", "id": "ffc980d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Bayes' theorem comes to our rescue here since (omitting the normalization constant)" ] @@ -4020,9 +3490,7 @@ { "cell_type": "markdown", "id": "860cdf65", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n", @@ -4032,9 +3500,7 @@ { "cell_type": "markdown", "id": "8ad984c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!" ] @@ -4042,9 +3508,7 @@ { "cell_type": "markdown", "id": "d0bee78c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and Bayes\n", "\n", @@ -4058,9 +3522,7 @@ { "cell_type": "markdown", "id": "bf039e41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -4070,9 +3532,7 @@ { "cell_type": "markdown", "id": "fffb60aa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -4080,9 +3540,7 @@ { "cell_type": "markdown", "id": "67b49eab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -4092,9 +3550,7 @@ { "cell_type": "markdown", "id": "cae7309e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", "did for OLS, this is most conveniently done by taking the negative\n", @@ -4105,9 +3561,7 @@ { "cell_type": "markdown", "id": "e15e70d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -4117,9 +3571,7 @@ { "cell_type": "markdown", "id": "36d005b5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/2\\tau^2$ with $\\lambda$ we have" ] @@ -4127,9 +3579,7 @@ { "cell_type": "markdown", "id": "4deb0ca9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -4139,9 +3589,7 @@ { "cell_type": "markdown", "id": "84fcc55c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Ridge cost function! Nice, isn't it?" ] @@ -4149,9 +3597,7 @@ { "cell_type": "markdown", "id": "47d7c3ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso and Bayes\n", "\n", @@ -4161,9 +3607,7 @@ { "cell_type": "markdown", "id": "8e745a19", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -4173,9 +3617,7 @@ { "cell_type": "markdown", "id": "55142cc6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -4183,9 +3625,7 @@ { "cell_type": "markdown", "id": "f627b93a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -4195,9 +3635,7 @@ { "cell_type": "markdown", "id": "1a1bae81", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the negative\n", "logarithm of the posterior probability and leaving out the\n", @@ -4207,9 +3645,7 @@ { "cell_type": "markdown", "id": "0c8a5a7b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -4219,9 +3655,7 @@ { "cell_type": "markdown", "id": "05bcb4ee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/\\tau$ with $\\lambda$ we have" ] @@ -4229,9 +3663,7 @@ { "cell_type": "markdown", "id": "308e688d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -4241,9 +3673,7 @@ { "cell_type": "markdown", "id": "e127394e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Lasso cost function!" ] @@ -4251,9 +3681,7 @@ { "cell_type": "markdown", "id": "ca26f6f7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exercise 1: mean values and variances in linear regression\n", "\n", @@ -4267,9 +3695,7 @@ { "cell_type": "markdown", "id": "a0f78141", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -4279,9 +3705,7 @@ { "cell_type": "markdown", "id": "a285460c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we minimized $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, with" @@ -4290,9 +3714,7 @@ { "cell_type": "markdown", "id": "f5f96083", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -4302,9 +3724,7 @@ { "cell_type": "markdown", "id": "21e62126", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The matrix $\\boldsymbol{X}$ is the so-called design matrix." ] @@ -4312,9 +3732,7 @@ { "cell_type": "markdown", "id": "e5730a62", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**a)**\n", "Show that the expectation value of $\\boldsymbol{y}$ for a given element $i$" @@ -4323,9 +3741,7 @@ { "cell_type": "markdown", "id": "55b4af74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -4337,9 +3753,7 @@ { "cell_type": "markdown", "id": "c4fb720a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and that\n", "its variance is" @@ -4348,9 +3762,7 @@ { "cell_type": "markdown", "id": "31a32444", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\sigma^2. \n", @@ -4361,9 +3773,7 @@ { "cell_type": "markdown", "id": "18736ca9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$." @@ -4372,9 +3782,7 @@ { "cell_type": "markdown", "id": "889dcad9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**b)**\n", "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ show that" @@ -4383,9 +3791,7 @@ { "cell_type": "markdown", "id": "65d7114c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\beta}) = \\boldsymbol{\\beta}.\n", @@ -4395,9 +3801,7 @@ { "cell_type": "markdown", "id": "0246cb71", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**c)**\n", "Show finally that the variance of $\\boldsymbol{\\beta}$ is" @@ -4406,9 +3810,7 @@ { "cell_type": "markdown", "id": "9a239b08", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -4420,9 +3822,7 @@ { "cell_type": "markdown", "id": "9b868ddb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exercise 2: Adding Ridge and Lasso Regression\n", "\n", @@ -4446,10 +3846,7 @@ "cell_type": "code", "execution_count": 12, "id": "fe00295c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = np.random.rand(100)\n", @@ -4459,9 +3856,7 @@ { "cell_type": "markdown", "id": "d59b60b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**a)**\n", "Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\\lambda$. Study the dependence on $\\lambda$ while also varying the strength of the noise in your expression for $y(x)$." @@ -4470,16 +3865,32 @@ { "cell_type": "markdown", "id": "47dc99e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**b)**\n", "Our next step is to study the variance of the parameters $\\beta_1$ and $\\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using **Scikit-Learn** to find the parameters $\\beta$. From your results calculate the variance of these parameters (recall that this is equal to the diagonal elements of the matrix $(\\hat{X}^T\\hat{X})+\\lambda\\hat{I})^{-1}$). Discuss the results of these variances as functions of $\\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figures 3.10 and 3.11. **Scikit-Learn** may not provide the variance of the parameters $\\beta$. This needs to be checked. With your own code you can however do so." ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.14" + } + }, "nbformat": 4, "nbformat_minor": 5 }