lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:00:16 +02:00
parent 1af325e7fb
commit 30d664a82b
9 changed files with 2392 additions and 942 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/chapter2.ipynb
+84 -26
View File
@@ -13,7 +13,7 @@
"What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n",
"\n",
"\n",
"We have shown that in ordinary least squares the optimal parameters $\\beta$ are given by"
"We have shown that in ordinary least squares (OLS) the optimal parameters $\\beta$ are given by"
]
},
{
@@ -21,7 +21,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
@@ -79,7 +79,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a projection matrix.\n",
"The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a [projection matrix](https://en.wikipedia.org/wiki/Projection_matrix).\n",
"We can then interpret our optimal model $\\tilde{\\boldsymbol{y}}$ as being represented by an orthogonal projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$. In our case here the matrix $\\boldsymbol{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.\n",
"\n",
"\n",
@@ -545,7 +545,7 @@
"\n",
"## Code for SVD and Inversion of Matrices\n",
"\n",
"How do we use the SVD to invert a matrix $\\boldsymbol{X}^\\boldsymbol{X}$ which is singular or near singular?\n",
"How do we use the SVD to invert a matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ which is singular or near singular?\n",
"The simple answer is to use the linear algebra function for the pseudoinverse, that is"
]
},
@@ -3258,6 +3258,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**More text will be added to this example.**\n",
"\n",
"## Linking the regression analysis with a statistical interpretation\n",
"\n",
"We will now couple the discussions of ordinary least squares, Ridge\n",
@@ -3624,7 +3626,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$.\n",
"It is a conditional probability (see below) and reads as the\n",
"likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters\n",
"$\\boldsymbol{\\beta}$.\n",
"\n",
"\n",
"In statistics, maximum likelihood estimation (MLE) is a method of\n",
@@ -3728,7 +3732,7 @@
"We define also the likelihood for $X$ and $Y$ as $p(X)$ and $p(Y)$ respectively.\n",
"The likelihood of a specific event $x_i$ (or $y_i$) is then written as $p(X=x_i)$ or just $p(x_i)=p_i$. \n",
"\n",
"**Union of events is given by.**"
"The union of events is given by"
]
},
{
@@ -3744,7 +3748,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**The product rule (aka joint probability) is given by.**"
"The product rule (aka joint probability) is given by"
]
},
{
@@ -3762,8 +3766,6 @@
"source": [
"where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n",
"\n",
"\n",
"\n",
"If we have independent events then $p(X,Y)=p(X)p(Y)$.\n",
"\n",
"\n",
@@ -3840,21 +3842,27 @@
"the parameter space represented by $X$. This function is not\n",
"necesseraly normalized and is normally called the likelihood function.\n",
"\n",
"The function $p(X)$ on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.\n",
"The function $p(X)$ on the right hand side is called the prior while\n",
"the function on the left hand side is the called the posterior\n",
"probability. The denominator on the right hand side serves as a\n",
"normalization factor for the posterior distribution.\n",
"\n",
"Let us try to illustrate Bayes' theorem through an example.\n",
"\n",
"\n",
"Let us suppose that you are undergoing a series of mammography scans in\n",
"order to rule out possible breast cancer cases. We define the\n",
"Let us suppose that you are undergoing a series of mammography scans\n",
"in order to rule out possible breast cancer cases. We define the\n",
"sensitivity for a positive event by the variable $X$. It takes binary\n",
"values with $X=1$ representing a positive event and $X=0$ being a\n",
"negative event. We reserve $Y$ as a classification parameter for\n",
"either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing).\n",
"either a negative or a positive breast cancer confirmation. (Short\n",
"note on wordings: positive here means having breast cancer, although\n",
"none of us would consider this being a positive thing).\n",
"\n",
"We let $Y=1$ represent the the case of having breast cancer and $Y=0$ as not.\n",
"\n",
"Let us assume that if you have breast cancer, the test will be positive with a probability of $0.8$, that is we have"
"Let us assume that if you have breast cancer, the test will be positive with a probability of $0.8$ (the numbers here are all made up),\n",
"that is we have"
]
},
{
@@ -3870,12 +3878,32 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of $80\\%$ for having cancer.\n",
"It is however not correct, as the following Bayesian analysis shows.\n",
"This obviously sounds scary since many would conclude that if the test\n",
"is positive, there is a likelihood of $80\\%$ for having cancer. It is\n",
"however not correct, as the following Bayesian analysis shows. The correct question to pose is *what is the probability of having breast cancer in case of a positive test?*\n",
"We are thus interested in"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"p(Y=1\\vert X=1),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"instead of $p(X=1\\vert Y=1)$.\n",
"\n",
"\n",
"If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number.\n",
"Let us assume that the prior probability in the population as a whole is"
"\n",
"If we look at various national surveys on breast cancer, the general\n",
"likelihood of developing breast cancer is a very small number. Let us\n",
"assume that the prior probability in the population as a whole is"
]
},
{
@@ -3891,7 +3919,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have"
"We need also to account for the fact that the test may produce a false\n",
"positive result (false alarm). Let us here assume that we have"
]
},
{
@@ -3907,15 +3936,38 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute"
"Using Bayes' theorem we can then find the posterior probability that\n",
"the person has breast cancer in case of a positive test, that is we\n",
"can compute"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"_auto2\"></div>\n",
"\n",
"$$\n",
"p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}=\\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n",
"\\begin{equation}\n",
"p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}= \n",
"\\label{_auto2} \\tag{2}\n",
"\\end{equation}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"_auto3\"></div>\n",
"\n",
"$$\n",
"\\begin{equation} \n",
" \\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n",
"\\label{_auto3} \\tag{3}\n",
"\\end{equation}\n",
"$$"
]
},
@@ -3941,9 +3993,12 @@
"parameters $\\boldsymbol{\\beta}$ using OLS, Ridge and Lasso regression. For\n",
"OLS, you will notice as function of the noise and polynomial degree,\n",
"that the parameters $\\beta$ will fluctuate from order to order in the\n",
"polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS.\n",
"polynomial fit and that for larger and larger polynomial degrees of\n",
"freedom, the parameters will tend to increase in value for OLS.\n",
"\n",
"For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one."
"For Ridge and Lasso regression, the higher order parameters will\n",
"typically be reduced, providing thereby less fluctuations from one\n",
"order to another one."
]
},
{
@@ -4008,8 +4063,8 @@
" # Compute the MSE and print it\n",
" MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n",
" MSELassoPredict[i] = MSE(y_test,ypredictLasso)\n",
" print(lmb,RegRidge.coef_)\n",
" print(lmb,RegLasso.coef_)\n",
"# print(lmb,RegRidge.coef_)\n",
"# print(lmb,RegLasso.coef_)\n",
"# Now plot the results\n",
"plt.figure()\n",
"plt.plot(np.log10(lambdas), MSERidgePredict, 'b', label = 'MSE Ridge Test')\n",
@@ -4024,10 +4079,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"How can we understand this? \n",
"How can we understand this? **More text to be added**.\n",
"\n",
"\n",
"\n",
"## Linking Bayes' Theorem with Ridge and Lasso Regression\n",
"\n",
"\n",
"Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression. \n",
"\n",
"For ordinary least squares we postulated that the maximum likelihood for the doamin of events $\\boldsymbol{D}$ (one-dimensional case)"