lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on book

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-08-27 07:15:11 +02:00
parent d35fa373a3
commit 747293f0d3
36 changed files with 393 additions and 381 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/_images/chapter1_11_0.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 6.2 KiB

After

Width:  |  Height:  |  Size: 6.5 KiB

doc/LectureNotes/_build/html/_images/chapter1_13_1.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 9.4 KiB

After

Width:  |  Height:  |  Size: 9.2 KiB

doc/LectureNotes/_build/html/_images/chapter1_27_0.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 13 KiB

doc/LectureNotes/_build/html/_images/chapter1_3_0.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 9.9 KiB

After

Width:  |  Height:  |  Size: 9.4 KiB

doc/LectureNotes/_build/html/_images/chapter1_61_10.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

doc/LectureNotes/_build/html/_images/chapter2_25_2.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.7 KiB

After

Width:  |  Height:  |  Size: 4.6 KiB

doc/LectureNotes/_build/html/_sources/chapter1.ipynb
+14 -14
View File
@@ -239,7 +239,7 @@
"\n",
"What follows is a simple Python code where we have defined a function\n",
"$y$ in terms of the variable $x$. Both are defined as vectors with $100$ entries. \n",
"The numbers in the vector $\\hat{x}$ are given\n",
"The numbers in the vector $\\boldsymbol{x}$ are given\n",
"by random numbers generated with a uniform distribution with entries\n",
"$x_i \\in [0,1]$ (more about probability distribution functions\n",
"later). These values are then used to define a function $y(x)$\n",
@@ -275,7 +275,7 @@
"distribution. From **Scikit-Learn** we import then the\n",
"**LinearRegression** functionality and make a prediction $\\tilde{y} =\n",
"\\alpha + \\beta x$ using the function **fit(x,y)**. We call the set of\n",
"data $(\\hat{x},\\hat{y})$ for our training data. The Python package\n",
"data $(\\boldsymbol{x},\\boldsymbol{y})$ for our training data. The Python package\n",
"**scikit-learn** has also a functionality which extracts the above\n",
"fitting parameters $\\alpha$ and $\\beta$ (see below). Later we will\n",
"distinguish between training data and test data.\n",
@@ -407,7 +407,7 @@
"metadata": {},
"source": [
"$$\n",
"\\epsilon_{\\mathrm{relative}}= \\frac{\\vert \\hat{y} -\\hat{\\tilde{y}}\\vert}{\\vert \\hat{y}\\vert}.\n",
"\\epsilon_{\\mathrm{relative}}= \\frac{\\vert \\boldsymbol{y} -\\boldsymbol{\\tilde{y}}\\vert}{\\vert \\boldsymbol{y}\\vert}.\n",
"$$"
]
},
@@ -521,7 +521,7 @@
"metadata": {},
"source": [
"$$\n",
"MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n",
"MSE(\\boldsymbol{y},\\boldsymbol{\\tilde{y}}) = \\frac{1}{n}\n",
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
"$$"
]
@@ -538,10 +538,10 @@
"determination. It provides a measure of how well future samples are\n",
"likely to be predicted by the model. Best possible score is 1.0 and it\n",
"can be negative (because the model can be arbitrarily worse). A\n",
"constant model that always predicts the expected value of $\\hat{y}$,\n",
"constant model that always predicts the expected value of $\\boldsymbol{y}$,\n",
"disregarding the input features, would get a $R^2$ score of $0.0$.\n",
"\n",
"If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
"If $\\tilde{\\boldsymbol{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
]
},
{
@@ -549,7 +549,7 @@
"metadata": {},
"source": [
"$$\n",
"R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
"R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
"$$"
]
},
@@ -557,7 +557,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where we have defined the mean value of $\\hat{y}$ as"
"where we have defined the mean value of $\\boldsymbol{y}$ as"
]
},
{
@@ -583,7 +583,7 @@
"metadata": {},
"source": [
"$$\n",
"\\text{MAE}(\\hat{y}, \\hat{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n-1} \\left| y_i - \\tilde{y}_i \\right|.\n",
"\\text{MAE}(\\boldsymbol{y}, \\boldsymbol{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n-1} \\left| y_i - \\tilde{y}_i \\right|.\n",
"$$"
]
},
@@ -600,7 +600,7 @@
"metadata": {},
"source": [
"$$\n",
"\\text{MSLE}(\\hat{y}, \\hat{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n - 1} (\\log_e (1 + y_i) - \\log_e (1 + \\tilde{y}_i) )^2,\n",
"\\text{MSLE}(\\boldsymbol{y}, \\boldsymbol{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n - 1} (\\log_e (1 + y_i) - \\log_e (1 + \\tilde{y}_i) )^2,\n",
"$$"
]
},
@@ -3314,7 +3314,7 @@
"metadata": {},
"source": [
"$$\n",
"MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n",
"MSE(\\boldsymbol{y},\\boldsymbol{\\tilde{y}}) = \\frac{1}{n}\n",
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
"$$"
]
@@ -3324,7 +3324,7 @@
"metadata": {},
"source": [
"and the $R^2$ score function.\n",
"If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
"If $\\tilde{\\boldsymbol{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
]
},
{
@@ -3332,7 +3332,7 @@
"metadata": {},
"source": [
"$$\n",
"R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
"R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
"$$"
]
},
@@ -3340,7 +3340,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where we have defined the mean value of $\\hat{y}$ as"
"where we have defined the mean value of $\\boldsymbol{y}$ as"
]
},
{
doc/LectureNotes/_build/html/_sources/chapter2.ipynb
-3
View File
@@ -6,9 +6,6 @@
"source": [
"# Resampling Methods\n",
"\n",
"[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage)\n",
"\n",
"\n",
"## Introduction\n",
"\n",
"Resampling methods are an indispensable tool in modern\n",
doc/LectureNotes/_build/html/_sources/chapter4.ipynb
+32 -32
View File
@@ -16,11 +16,11 @@
"coefficients of a functional fit (say a polynomial) in order to be\n",
"able to predict the response of a continuous variable on some unseen\n",
"data. The fit to the continuous variable $y_i$ is based on some\n",
"independent variables $\\hat{x}_i$. Linear regression resulted in\n",
"independent variables $x_i$. Linear regression resulted in\n",
"analytical expressions for standard ordinary Least Squares or Ridge\n",
"regression (in terms of matrices to invert) for several quantities,\n",
"ranging from the variance and thereby the confidence intervals of the\n",
"parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
"optimal parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
"the product of the design matrices, linear regression gives then a\n",
"simple recipe for fitting our data.\n",
"\n",
@@ -65,7 +65,7 @@
"from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n",
"\n",
"The goal is to predict the\n",
"output classes from the design matrix $\\hat{X}\\in\\mathbb{R}^{n\\times p}$\n",
"output classes from the design matrix $\\boldsymbol{X}\\in\\mathbb{R}^{n\\times p}$\n",
"made of $n$ samples, each of which carries $p$ features or predictors. The\n",
"primary goal is to identify the classes to which new unseen samples\n",
"belong.\n",
@@ -107,7 +107,7 @@
"\n",
"$$\n",
"\\begin{equation}\n",
"\\hat{y} = \\hat{X}^T\\hat{\\beta} + \\hat{\\epsilon},\n",
"\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n",
"\\label{_auto1} \\tag{1}\n",
"\\end{equation}\n",
"$$"
@@ -117,8 +117,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n",
"$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators/predictors.\n",
"where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
"$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n",
"\n",
"\n",
"The main problem with our function is that it takes values on the\n",
@@ -380,8 +380,8 @@
"source": [
"$$\n",
"\\begin{align*}\n",
"p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
"p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n",
"p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
"p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
"\\end{align*}\n",
"$$"
]
@@ -390,7 +390,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"\n",
"Note that we used"
]
@@ -400,7 +400,7 @@
"metadata": {},
"source": [
"$$\n",
"p(y_i=0\\vert x_i, \\hat{\\beta}) = 1-p(y_i=1\\vert x_i, \\hat{\\beta}).\n",
"p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
"$$"
]
},
@@ -422,7 +422,7 @@
"source": [
"$$\n",
"\\begin{align*}\n",
"P(\\mathcal{D}|\\hat{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\hat{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
"P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
"\\end{align*}\n",
"$$"
]
@@ -439,7 +439,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\hat{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
"$$"
]
},
@@ -455,7 +455,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"$$"
]
},
@@ -472,7 +472,7 @@
"metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\hat{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"$$"
]
},
@@ -484,7 +484,7 @@
"in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
"\n",
"\n",
"The cross entropy is a convex function of the weights $\\hat{\\beta}$ and,\n",
"The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n",
"therefore, any local minimizer is a global minimizer. \n",
"\n",
"\n",
@@ -497,7 +497,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
"$$"
]
},
@@ -513,7 +513,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
"$$"
]
},
@@ -521,9 +521,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Let us now define a vector $\\hat{y}$ with $n$ elements $y_i$, an\n",
"$n\\times p$ matrix $\\hat{X}$ which contains the $x_i$ values and a\n",
"vector $\\hat{p}$ of fitted probabilities $p(y_i\\vert x_i,\\hat{\\beta})$. We can rewrite in a more compact form the first\n",
"Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n",
"$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n",
"vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n",
"derivative of cost function as"
]
},
@@ -532,7 +532,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
"$$"
]
},
@@ -540,8 +540,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If we in addition define a diagonal matrix $\\hat{W}$ with elements \n",
"$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as"
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
]
},
{
@@ -549,7 +549,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
"$$"
]
},
@@ -565,7 +565,7 @@
"metadata": {},
"source": [
"$$\n",
"\\log{ \\frac{p(\\hat{\\beta}\\hat{x})}{1-p(\\hat{\\beta}\\hat{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
"\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
"$$"
]
},
@@ -573,7 +573,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we defined $\\hat{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\hat{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
"Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
]
},
{
@@ -581,7 +581,7 @@
"metadata": {},
"source": [
"$$\n",
"p(\\hat{\\beta}\\hat{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
"p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
"$$"
]
},
@@ -654,7 +654,7 @@
"multinomial logistic regression and linear discriminant analysis, the\n",
"input to the function is the result of $K$ distinct linear functions,\n",
"and the predicted probability for the $k$-th class given a sample\n",
"vector $\\hat{x}$ and a weighting vector $\\hat{\\beta}$ is (with two\n",
"vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n",
"predictors):"
]
},
@@ -1183,7 +1183,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have"
"Defining the Jacobian matrix $\\boldsymbol{J}$ we have"
]
},
{
@@ -1191,7 +1191,7 @@
"metadata": {},
"source": [
"$$\n",
"{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n",
"\\boldsymbol{J}=\\left( \\begin{array}{cc}\n",
" \\partial f_1/\\partial x_1 & \\partial f_1/\\partial x_2 \\\\\n",
" \\partial f_2/\\partial x_1 &\\partial f_2/\\partial x_2\n",
" \\end{array} \\right),\n",
@@ -1229,7 +1229,7 @@
"source": [
"$$\n",
"\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n",
" -{\\bf \\boldsymbol{J}}^{-1}\n",
" -\\boldsymbol{J}^{-1}\n",
" \\left(\\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\\\ f_2(x_1^{n},x_2^{n}) \\end{array} \\right).\n",
"$$"
]
@@ -1240,7 +1240,7 @@
"source": [
"We need thus to compute the inverse of the Jacobian matrix and it\n",
"is to understand that difficulties may\n",
"arise in case ${\\bf \\boldsymbol{J}}$ is nearly singular.\n",
"arise in case $\\boldsymbol{J}$ is nearly singular.\n",
"\n",
"It is rather straightforward to extend the above scheme to systems of\n",
"more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. \n",
doc/LectureNotes/_build/html/chapter1.html
+56 -50
View File
@@ -588,7 +588,7 @@ may first try the simplest class of models, namely linear models, followed obvio
<p>We start with perhaps our simplest possible example, using <strong>Scikit-Learn</strong> to perform linear regression analysis on a data set produced by us.</p>
<p>What follows is a simple Python code where we have defined a function
<span class="math notranslate nohighlight">\(y\)</span> in terms of the variable <span class="math notranslate nohighlight">\(x\)</span>. Both are defined as vectors with <span class="math notranslate nohighlight">\(100\)</span> entries.
The numbers in the vector <span class="math notranslate nohighlight">\(\hat{x}\)</span> are given
The numbers in the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> are given
by random numbers generated with a uniform distribution with entries
<span class="math notranslate nohighlight">\(x_i \in [0,1]\)</span> (more about probability distribution functions
later). These values are then used to define a function <span class="math notranslate nohighlight">\(y(x)\)</span>
@@ -611,7 +611,7 @@ y = 2x+N(0,1),
distribution. From <strong>Scikit-Learn</strong> we import then the
<strong>LinearRegression</strong> functionality and make a prediction <span class="math notranslate nohighlight">\(\tilde{y} =
\alpha + \beta x\)</span> using the function <strong>fit(x,y)</strong>. We call the set of
data <span class="math notranslate nohighlight">\((\hat{x},\hat{y})\)</span> for our training data. The Python package
data <span class="math notranslate nohighlight">\((\boldsymbol{x},\boldsymbol{y})\)</span> for our training data. The Python package
<strong>scikit-learn</strong> has also a functionality which extracts the above
fitting parameters <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> (see below). Later we will
distinguish between training data and test data.</p>
@@ -703,7 +703,7 @@ the <span class="math notranslate nohighlight">\(\chi^2\)</span> function becom
the relative error (why would we prefer the MSE instead of the relative error?) as</p>
<div class="math notranslate nohighlight">
\[
\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
\epsilon_{\mathrm{relative}}= \frac{\vert \boldsymbol{y} -\boldsymbol{\tilde{y}}\vert}{\vert \boldsymbol{y}\vert}.
\]</div>
<p>The squared cost function results in an arithmetic mean-unbiased
estimator, and the absolute-value cost function results in a
@@ -781,13 +781,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha:
[1.98452685]
[2.15024669]
Coefficient beta :
[[5.00109273]]
Mean squared error: 0.19
Variance score: 0.92
[[4.89975818]]
Mean squared error: 0.26
Variance score: 0.90
Mean squared log error: 0.01
Mean absolute error: 0.35
Mean absolute error: 0.42
</pre></div>
</div>
<img alt="_images/chapter1_13_1.png" src="_images/chapter1_13_1.png" />
@@ -797,7 +797,7 @@ Mean absolute error: 0.35
<span class="math notranslate nohighlight">\(\alpha\)</span>. Depending on the constant in front of the normal distribution, we get values near or far from <span class="math notranslate nohighlight">\(alpha =2\)</span> and <span class="math notranslate nohighlight">\(\beta =5\)</span>. Try to play around with different parameters in front of the normal distribution. The function <strong>meansquarederror</strong> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as</p>
<div class="math notranslate nohighlight">
\[
MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
\]</div>
<p>The smaller the value, the better the fit. Ideally we would like to
@@ -807,14 +807,14 @@ this function as being similar to the <span class="math notranslate nohighlight"
determination. It provides a measure of how well future samples are
likely to be predicted by the model. Best possible score is 1.0 and it
can be negative (because the model can be arbitrarily worse). A
constant model that always predicts the expected value of <span class="math notranslate nohighlight">\(\hat{y}\)</span>,
constant model that always predicts the expected value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>,
disregarding the input features, would get a <span class="math notranslate nohighlight">\(R^2\)</span> score of <span class="math notranslate nohighlight">\(0.0\)</span>.</p>
<p>If <span class="math notranslate nohighlight">\(\tilde{\hat{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
<p>If <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
<div class="math notranslate nohighlight">
\[
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
\]</div>
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\hat{y}\)</span> as</p>
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
@@ -824,13 +824,13 @@ the mean absolute error (MAE), a risk metric corresponding to the expected value
The MAE is defined as follows</p>
<div class="math notranslate nohighlight">
\[
\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
\text{MAE}(\boldsymbol{y}, \boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
\]</div>
<p>We present the
squared logarithmic (quadratic) error</p>
<div class="math notranslate nohighlight">
\[
\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
\text{MSLE}(\boldsymbol{y}, \boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\log_e (x)\)</span> stands for the natural logarithm of <span class="math notranslate nohighlight">\(x\)</span>. This error
estimate is best to use when targets having exponential growth, such
@@ -888,7 +888,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter1_27_0.png" src="_images/chapter1_27_0.png" />
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.004999999999999996
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.0050000000000000044
</pre></div>
</div>
</div>
@@ -1225,7 +1225,7 @@ A
270 3344 160 110 270 Ds 7.253775 7.253775
[267 rows x 6 columns]
0.009883615646716184
0.009883615646716182
</pre></div>
</div>
</div>
@@ -1283,8 +1283,6 @@ functionality.</p>
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
@@ -1307,6 +1305,14 @@ functionality.</p>
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
@@ -1331,14 +1337,14 @@ functionality.</p>
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
@@ -1349,7 +1355,7 @@ functionality.</p>
warnings.warn(
</pre></div>
</div>
<img alt="_images/chapter1_61_9.png" src="_images/chapter1_61_9.png" />
<img alt="_images/chapter1_61_10.png" src="_images/chapter1_61_10.png" />
</div>
</div>
</div>
@@ -2324,13 +2330,13 @@ but now splitting the data into a training set and a test set.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
0.9999864543345858
0.9999868619217517
Training MSE
6.180092462880674
5.965885569080809
Test R2
0.9999822527140678
0.9999794306626945
Test MSE
7.205466494327873
8.300162456113691
</pre></div>
</div>
</div>
@@ -2741,31 +2747,31 @@ techniques.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
R2 score before scaling 0.99
R2 score before scaling 1.00
Feature min values before scaling:
[1.00000000e+00 1.10094646e-03 9.51523276e-04 1.21208310e-06
1.04757618e-06 9.05396545e-07 1.33443859e-09 1.15332528e-09
9.96793117e-10 8.61505887e-10 1.46914544e-12 1.26974938e-12
1.09741585e-12 9.48471852e-13 8.19742904e-13 1.61745046e-15
1.39792608e-15 1.20819609e-15 1.04421672e-15 9.02493044e-16
7.80004454e-16]
[1.00000000e+00 2.54152940e-03 1.38207279e-03 6.45937170e-06
3.51257863e-06 1.91012519e-06 1.64166831e-08 8.92732185e-09
4.85463934e-09 2.63993205e-09 4.17234827e-11 2.26890510e-11
1.23382086e-11 6.70946493e-12 3.64857826e-12 1.06041458e-13
5.76648901e-14 3.13579199e-14 1.70523024e-14 9.27296891e-15
5.04260073e-15]
Feature max values before scaling:
[1. 0.99825997 0.99883879 0.99652296 0.99710078 0.99767893
0.99478898 0.9953658 0.99594294 0.99652042 0.99305802 0.99363383
0.99420997 0.99478645 0.99536326 0.99133007 0.99190487 0.99248001
0.99305549 0.99363129 0.99420743]
[1. 0.99817842 0.99945628 0.99636015 0.99763569 0.99891285
0.9945452 0.99581841 0.99709325 0.99836973 0.99273355 0.99400444
0.99527696 0.99655111 0.99782689 0.9909252 0.99219378 0.99346398
0.99473581 0.99600927 0.99728435]
Feature min values after scaling:
[ 0. -1.61869821 -1.66880047 -1.06209126 -1.08170444 -1.10131725
-0.84087101 -0.85396354 -0.86692943 -0.87972591 -0.71351486 -0.7240496
-0.73453972 -0.74495014 -0.75524378 -0.62783293 -0.63680118 -0.64580686
-0.65482578 -0.66383151 -0.67279536]
[ 0. -1.63437572 -1.76504618 -1.03871832 -1.08415761 -1.13457922
-0.81620806 -0.83935285 -0.86436607 -0.8914984 -0.69347005 -0.70790937
-0.72312577 -0.73921714 -0.75629493 -0.61234223 -0.6226921 -0.63339159
-0.64447921 -0.65599927 -0.66800261]
Feature max values after scaling:
[0. 1.78944806 1.68342382 2.34172919 2.25457052 2.16553696
2.7899453 2.71281409 2.63374631 2.55280484 3.17117385 3.10307631
3.03303359 2.96104648 2.88712946 3.50394742 3.44395541 3.38216436
3.31853484 3.25303483 3.1856411 ]
[0. 1.86574276 1.72218808 2.41770932 2.31730641 2.21347282
2.88297395 2.79328828 2.70234019 2.60999846 3.291614 3.20772452
3.12283463 3.03697069 2.95014575 3.65766387 3.57871326 3.49865673
3.41754964 3.33544681 3.25240108]
MSE after scaling: 0.00
R2 score for scaled data: 0.99
R2 score for scaled data: 1.00
</pre></div>
</div>
</div>
@@ -2849,16 +2855,16 @@ The following simple Python instructions define our <span class="math notranslat
</ol>
<div class="math notranslate nohighlight">
\[
MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
\]</div>
<p>and the <span class="math notranslate nohighlight">\(R^2\)</span> score function.
If <span class="math notranslate nohighlight">\(\tilde{\hat{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
If <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
<div class="math notranslate nohighlight">
\[
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
\]</div>
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\hat{y}\)</span> as</p>
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
doc/LectureNotes/_build/html/chapter2.html
+7 -8
View File
@@ -55,7 +55,7 @@
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="5. Ridge and Lasso Regression" href="chapter3.html" />
<link rel="prev" title="3. Linear Regression, basic Elements" href="chapter1.html" />
<link rel="prev" title="3. Linear Regression" href="chapter1.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
@@ -139,7 +139,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression, basic Elements
3. Linear Regression
</a>
</li>
<li class="toctree-l1 current active">
@@ -329,7 +329,6 @@
<div class="section" id="resampling-methods">
<h1><span class="section-number">4. </span>Resampling Methods<a class="headerlink" href="#resampling-methods" title="Permalink to this headline"></a></h1>
<p><a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage">Video of Lecture</a></p>
<div class="section" id="introduction">
<h2><span class="section-number">4.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline"></a></h2>
<p>Resampling methods are an indispensable tool in modern
@@ -588,10 +587,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.137268 sec
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.139475 sec
Jackknife Statistics :
original bias std. error
99.8901 99.8802 0.152636
99.6714 99.6614 0.149667
</pre></div>
</div>
</div>
@@ -719,10 +718,10 @@ theorem.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 1.80121 sec
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 1.78195 sec
Bootstrap Statistics :
original bias std. error
99.7162 15.0832 99.7172 0.149366
100.073 14.9354 100.074 0.149903
</pre></div>
</div>
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
@@ -1408,7 +1407,7 @@ flexible statistical methods have higher variance.</p>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter1.html" title="previous page"><span class="section-number">3. </span>Linear Regression, basic Elements</a>
<a class='left-prev' id="prev-link" href="chapter1.html" title="previous page"><span class="section-number">3. </span>Linear Regression</a>
<a class='right-next' id="next-link" href="chapter3.html" title="next page"><span class="section-number">5. </span>Ridge and Lasso Regression</a>
</div>
doc/LectureNotes/_build/html/chapter4.html
+33 -33
View File
@@ -139,7 +139,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression, basic Elements
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
@@ -582,11 +582,11 @@
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable <span class="math notranslate nohighlight">\(y_i\)</span> is based on some
independent variables <span class="math notranslate nohighlight">\(\hat{x}_i\)</span>. Linear regression resulted in
independent variables <span class="math notranslate nohighlight">\(x_i\)</span>. Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> to the mean squared error. If we can invert
optimal parameters <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.</p>
<p>Classification problems, however, are concerned with outcomes taking
@@ -622,7 +622,7 @@ models, as we will see later.</p>
responses or the outcomes, <span class="math notranslate nohighlight">\(y_i\)</span> are discrete and only take values
from <span class="math notranslate nohighlight">\(k=0,\dots,K-1\)</span> (i.e. <span class="math notranslate nohighlight">\(K\)</span> classes).</p>
<p>The goal is to predict the
output classes from the design matrix <span class="math notranslate nohighlight">\(\hat{X}\in\mathbb{R}^{n\times p}\)</span>
output classes from the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in\mathbb{R}^{n\times p}\)</span>
made of <span class="math notranslate nohighlight">\(n\)</span> samples, each of which carries <span class="math notranslate nohighlight">\(p\)</span> features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.</p>
@@ -645,12 +645,12 @@ weighted linear combination, namely</p>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} + \boldsymbol{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
\]</div>
<p>where <span class="math notranslate nohighlight">\(\hat{y}\)</span> is a vector representing the possible outcomes, <span class="math notranslate nohighlight">\(\hat{X}\)</span> is our
<span class="math notranslate nohighlight">\(n\times p\)</span> design matrix and <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> represents our estimators/predictors.</p>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> is a vector representing the possible outcomes, <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is our
<span class="math notranslate nohighlight">\(n\times p\)</span> design matrix and <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> represents our estimators/predictors.</p>
<p>The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels <span class="math notranslate nohighlight">\(y_i\)</span> are discrete variables. A typical example is the credit
@@ -857,15 +857,15 @@ p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &amp;= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &amp;= 1 - p(y_i=1|x_i,\hat{\beta}),
p(y_i=1|x_i,\boldsymbol{\beta}) &amp;= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\beta}) &amp;= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
\end{align*}
\end{split}\]</div>
<p>where <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>.</p>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>.</p>
<p>Note that we used</p>
<div class="math notranslate nohighlight">
\[
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
\]</div>
<p>In order to define the total likelihood for all possible outcomes from a<br />
dataset <span class="math notranslate nohighlight">\(\mathcal{D}=\{(y_i,x_i)\}\)</span>, with the binary labels
@@ -876,63 +876,63 @@ likelihood in terms of the product of the individual probabilities of a specific
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
P(\mathcal{D}|\hat{\beta})&amp; = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
P(\mathcal{D}|\boldsymbol{\beta})&amp; = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
\end{split}\]</div>
<p>from which we obtain the log-likelihood and our <strong>cost/loss</strong> function</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
\]</div>
<p>Reordering the logarithms, we can rewrite the <strong>cost/loss</strong> function as</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\]</div>
<p>The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to <span class="math notranslate nohighlight">\(\beta\)</span>.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\]</div>
<p>This equation is known in statistics as the <strong>cross entropy</strong>. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually <span class="math notranslate nohighlight">\(L_1\)</span> and <span class="math notranslate nohighlight">\(L_2\)</span> regularization as we did for Ridge and Lasso regression.</p>
<p>The cross entropy is a convex function of the weights <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> and,
<p>The cross entropy is a convex function of the weights <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and,
therefore, any local minimizer is a global minimizer.</p>
<p>Minimizing this
cost function with respect to the two parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\]</div>
<p>Let us now define a vector <span class="math notranslate nohighlight">\(\hat{y}\)</span> with <span class="math notranslate nohighlight">\(n\)</span> elements <span class="math notranslate nohighlight">\(y_i\)</span>, an
<span class="math notranslate nohighlight">\(n\times p\)</span> matrix <span class="math notranslate nohighlight">\(\hat{X}\)</span> which contains the <span class="math notranslate nohighlight">\(x_i\)</span> values and a
vector <span class="math notranslate nohighlight">\(\hat{p}\)</span> of fitted probabilities <span class="math notranslate nohighlight">\(p(y_i\vert x_i,\hat{\beta})\)</span>. We can rewrite in a more compact form the first
<p>Let us now define a vector <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with <span class="math notranslate nohighlight">\(n\)</span> elements <span class="math notranslate nohighlight">\(y_i\)</span>, an
<span class="math notranslate nohighlight">\(n\times p\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which contains the <span class="math notranslate nohighlight">\(x_i\)</span> values and a
vector <span class="math notranslate nohighlight">\(\boldsymbol{p}\)</span> of fitted probabilities <span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\beta})\)</span>. We can rewrite in a more compact form the first
derivative of cost function as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
\]</div>
<p>If we in addition define a diagonal matrix <span class="math notranslate nohighlight">\(\hat{W}\)</span> with elements
<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})\)</span>, we can obtain a compact expression of the second derivative as</p>
<p>If we in addition define a diagonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{W}\)</span> with elements
<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta})\)</span>, we can obtain a compact expression of the second derivative as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
\]</div>
<p>Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with <span class="math notranslate nohighlight">\(p\)</span> predictors</p>
<div class="math notranslate nohighlight">
\[
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\]</div>
<p>Here we defined <span class="math notranslate nohighlight">\(\hat{x}=[1,x_1,x_2,\dots,x_p]\)</span> and <span class="math notranslate nohighlight">\(\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]\)</span> leading to</p>
<p>Here we defined <span class="math notranslate nohighlight">\(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p]\)</span> leading to</p>
<div class="math notranslate nohighlight">
\[
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
\]</div>
<p>Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to <span class="math notranslate nohighlight">\(K\)</span> classes. Let us for the sake
@@ -962,7 +962,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of <span class="math notranslate nohighlight">\(K\)</span> distinct linear functions,
and the predicted probability for the <span class="math notranslate nohighlight">\(k\)</span>-th class given a sample
vector <span class="math notranslate nohighlight">\(\hat{x}\)</span> and a weighting vector <span class="math notranslate nohighlight">\(\hat{\beta}\)</span> is (with two
vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> and a weighting vector <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is (with two
predictors):</p>
<div class="math notranslate nohighlight">
\[
@@ -1279,10 +1279,10 @@ and variables. Consider the case with two equations</p>
\partial f_2/\partial x_2+\dots
\end{array}.
\end{split}\]</div>
<p>Defining the Jacobian matrix <span class="math notranslate nohighlight">\({\bf \boldsymbol{J}}\)</span> we have</p>
<p>Defining the Jacobian matrix <span class="math notranslate nohighlight">\(\boldsymbol{J}\)</span> we have</p>
<div class="math notranslate nohighlight">
\[\begin{split}
{\bf \boldsymbol{J}}=\left( \begin{array}{cc}
\boldsymbol{J}=\left( \begin{array}{cc}
\partial f_1/\partial x_1 &amp; \partial f_1/\partial x_2 \\
\partial f_2/\partial x_1 &amp;\partial f_2/\partial x_2
\end{array} \right),
@@ -1298,12 +1298,12 @@ and variables. Consider the case with two equations</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)=
-{\bf \boldsymbol{J}}^{-1}
-\boldsymbol{J}^{-1}
\left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right).
\end{split}\]</div>
<p>We need thus to compute the inverse of the Jacobian matrix and it
is to understand that difficulties may
arise in case <span class="math notranslate nohighlight">\({\bf \boldsymbol{J}}\)</span> is nearly singular.</p>
arise in case <span class="math notranslate nohighlight">\(\boldsymbol{J}\)</span> is nearly singular.</p>
<p>It is rather straightforward to extend the above scheme to systems of
more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function.</p>
</div>
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long