lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2111fd0fbcd18bebee148bb9af34ac585bcb62ae
FYS-STK4155/doc/pub/Regression/ipynb/Regression.ipynb
T
mhjensen d52541b08e Testing doconce with python3.6
2017-11-21 17:46:17 +01:00

922 lines
24 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- dom:TITLE: Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis -->\n",
"# Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis\n",
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Nov 21, 2017**\n",
"\n",
"Copyright 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"\n",
"## Regression analysis, overarching aims\n",
"\n",
"Regression modeling deals with the description of the sampling distribution of a given random variable $y$ varies as function of another variable or a set of such variables $\\hat{x} =[x_0, x_1,\\dots, x_p]^T$. \n",
"The first variable is called the **dependent**, the **outcome** or the **response** variable while the set of variables $\\hat{x}$ is called the independent variable, or the predictor variable or the explanatory variable. \n",
"\n",
"A regression model aims at finding a likelihood function $p(y\\vert \\hat{x})$, that is the conditional distribution for $y$ with a given $\\hat{x}$. The estimation of $p(y\\vert \\hat{x})$ is made using a data set with \n",
"* $n$ cases $i = 0, 1, 2, \\dots, n-1$ \n",
"\n",
"* Response (dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n",
"\n",
"* $p$ Explanatory (independent or predictor) variables $\\hat{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip}]$ with $i = 0, 1, 2, \\dots, n-1$ \n",
"\n",
" The goal of the regression analysis is to extract/exploit relationship between $y_i$ and $\\hat{x}_i$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions .\n",
"\n",
"\n",
"\n",
"\n",
"## General linear models\n",
"Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\\hat{y}=[y_0,y_1,\\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\\hat{x}=[x_0,x_1,\\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. \n",
"\n",
"Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_i x_i^j+\\epsilon_i,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\epsilon_i$ is the error in our approximation.\n",
"\n",
"\n",
"\n",
"\n",
"## Rewriting the fitting procedure as a linear algebra problem\n",
"For every set of values $y_i,x_i$ we have thus the corresponding set of equations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
"y_0&=\\beta_0+\\beta_1x_0^1+\\beta_2x_0^2+\\dots+\\beta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n",
"y_1&=\\beta_0+\\beta_1x_1^1+\\beta_2x_1^2+\\dots+\\beta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n",
"y_2&=\\beta_0+\\beta_1x_2^1+\\beta_2x_2^2+\\dots+\\beta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n",
"\\dots & \\dots \\\\\n",
"y_{n-1}&=\\beta_0+\\beta_1x_{n-1}^1+\\beta_2x_{n-1}^2+\\dots+\\beta_1x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Rewriting the fitting procedure as a linear algebra problem, follows\n",
"Defining the vectors"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2\n",
" \n",
"<\n",
"<\n",
"<\n",
"!\n",
"!\n",
"M\n",
"A\n",
"T\n",
"H\n",
"_\n",
"B\n",
"L\n",
"O\n",
"C\n",
"K"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3\n",
" \n",
"<\n",
"<\n",
"<\n",
"!\n",
"!\n",
"M\n",
"A\n",
"T\n",
"H\n",
"_\n",
"B\n",
"L\n",
"O\n",
"C\n",
"K"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\epsilon} = [\\epsilon_0,\\epsilon_1, \\epsilon_2,\\dots, \\epsilon_{n-1}]^T,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and the matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{X}=\n",
"\\begin{bmatrix} \n",
"1& x_{0}^1 &x_{0}^2& \\dots & \\dots &x_{0}^{n-1}\\\\\n",
"1& x_{1}^1 &x_{1}^2& \\dots & \\dots &x_{1}^{n-1}\\\\\n",
"1& x_{2}^1 &x_{2}^2& \\dots & \\dots &x_{2}^{n-1}\\\\ \n",
"\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
"1& x_{n-1}^1 &x_{n-1}^2& \\dots & \\dots &x_{n-1}^{n-1}\\\\\n",
"\\end{bmatrix}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we can rewrite our equations as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{y} = \\hat{X}\\hat{\\beta}+\\hat{\\epsilon}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generalizing the fitting procedure as a linear algebra problem\n",
"We are obviously not limited to the above polynomial. We could replace the various powers of $x$ with elements of Fourier series, that is, instead of $x_i^j$ we could have $\\cos{(j x_i)}$ or $\\sin{(j x_i)}$, or time series or other orthogonal functions.\n",
"For every set of values $y_i,x_i$ we can then generalize the equations to"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
"y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
"y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
"y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n",
"\\dots & \\dots \\\\\n",
"y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_i\\\\\n",
"\\dots & \\dots \\\\\n",
"y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_1x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generalizing the fitting procedure as a linear algebra problem\n",
"We redefine in turn the matrix $\\hat{X}$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{X}=\n",
"\\begin{bmatrix} \n",
"x_{00}& x_{01} &x_{02}& \\dots & \\dots &x_{0,n-1}\\\\\n",
"x_{10}& x_{11} &x_{12}& \\dots & \\dots &x_{1,n-1}\\\\\n",
"x_{20}& x_{21} &x_{22}& \\dots & \\dots &x_{2,n-1}\\\\ \n",
"\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
"x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,n-1}\\\\\n",
"\\end{bmatrix}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and without loss of generality we rewrite again our equations as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{y} = \\hat{X}\\hat{\\beta}+\\hat{\\epsilon}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The left-hand side of this equation forms know. Our error vector $\\hat{\\epsilon}$ and the parameter vector $\\hat{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values?\n",
"\n",
"\n",
"\n",
"\n",
"## Optimizing our parameters\n",
"We have defined the matrix $\\hat{X}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
"y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
"y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
"y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n",
"\\dots & \\dots \\\\\n",
"y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_1\\\\\n",
"\\dots & \\dots \\\\\n",
"y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_1x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Optimizing our parameters, more details\n",
"We well use this matrix to define the approximation $\\hat{\\tilde{y}}$ via the unknown quantity $\\hat{\\beta}$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\tilde{y}}= \\hat{X}\\hat{\\beta},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parametrized values $\\tilde{y}_i$, namely"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"Q(\\hat{\\beta})=\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\left(\\hat{y}-\\hat{\\tilde{y}}\\right)^T\\left(\\hat{y}-\\hat{\\tilde{y}}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or using the matrix $\\hat{X}$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"Q(\\hat{\\beta})=\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right)^T\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Interpretations and optimizing our parameters\n",
"The function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"Q(\\hat{\\beta})=\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right)^T\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that till now we have treated $y_i$ as the exact value. Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable.\n",
"\n",
"In order to find the parameters $\\beta_i$ we will then minimize the spread of $Q(\\hat{\\beta})$ by requiring"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which results in"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\beta_j} = -2\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or in a matrix-vector form as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Interpretations and optimizing our parameters\n",
"We can rewrite"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{X}^T\\hat{y} = \\hat{X}^T\\hat{X}\\hat{\\beta},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and if the matrix $\\hat{X}^T\\hat{X}$ is invertible we have the solution"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\beta} =\\left(\\hat{X}^T\\hat{X}\\right)^{-1}\\hat{X}^T\\hat{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Interpretations and optimizing our parameters\n",
"The residuals $\\hat{\\epsilon}$ are in turn given by"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\epsilon} = \\hat{y}-\\hat{\\tilde{y}} = \\hat{y}-\\hat{X}\\hat{\\beta},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and with"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right)= 0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{X}^T\\hat{\\epsilon}=\\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right)= 0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"meaning that the solution for $\\hat{\\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.\n",
"\n",
"\n",
"\n",
"## The $\\chi^2$ function\n",
"\n",
"Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable.\n",
"\n",
"Introducing the standard deviation $\\sigma_i$ for each measurement $y_i$, we define now the $\\chi^2$ function as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\chi^2(\\hat{\\beta})=\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\left(\\hat{y}-\\hat{\\tilde{y}}\\right)^T\\frac{1}{\\hat{\\Sigma^2}}\\left(\\hat{y}-\\hat{\\tilde{y}}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where the matrix $\\hat{\\Sigma}$ is a diagonal matrix with $\\sigma_i$ as matrix elements.\n",
"\n",
"\n",
"\n",
"## The $\\chi^2$ function\n",
"\n",
"In order to find the parameters $\\beta_i$ we will then minimize the spread of $\\chi^2(\\hat{\\beta})$ by requiring"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which results in"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_j} = -2\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or in a matrix-vector form as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{A}^T\\left( \\hat{b}-\\hat{A}\\hat{\\beta}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where we have defined the matrix $\\hat{A} =\\hat{X}/\\hat{\\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\\sigma_i$ and the vector $\\hat{b}$ with elements $b_i = y_i/\\sigma_i$.\n",
"\n",
"\n",
"\n",
"## The $\\chi^2$ function\n",
"\n",
"We can rewrite"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{A}^T\\left( \\hat{b}-\\hat{A}\\hat{\\beta}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{A}^T\\hat{b} = \\hat{A}^T\\hat{A}\\hat{\\beta},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and if the matrix $\\hat{A}^T\\hat{A}$ is invertible we have the solution"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\beta} =\\left(\\hat{A}^T\\hat{A}\\right)^{-1}\\hat{A}^T\\hat{b}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The $\\chi^2$ function\n",
"\n",
"If we then introduce the matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{H} = \\hat{A}^T\\hat{A},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we have then the following expression for the parameters $\\beta_j$ (the matrix elements of $\\hat{H}$ are $h_{ij}$)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We state without proof the expression for the uncertainty in the parameters $\\beta_j$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\sigma^2(\\beta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\beta_j}{\\partial y_i}\\right)^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"resulting in"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\sigma^2(\\beta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The $\\chi^2$ function\n",
"The first step here is to approximate the function $y$ with a first-order polynomial, that is we write"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y=y(x) \\rightarrow y(x_i) \\approx \\beta_0+\\beta_1 x_i.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By computing the derivatives of $\\chi^2$ with respect to $\\beta_0$ and $\\beta_1$ show that these are given by"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_0} = -2\\left[ \\sum_{i=0}^{1}\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_0} = -2\\left[ \\sum_{i=0}^{1}x_i\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The $\\chi^2$ function\n",
"\n",
"We define then"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\gamma = \\sum_{i=0}^{1}\\frac{1}{\\sigma_i^2},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4\n",
"0\n",
" \n",
"<\n",
"<\n",
"<\n",
"!\n",
"!\n",
"M\n",
"A\n",
"T\n",
"H\n",
"_\n",
"B\n",
"L\n",
"O\n",
"C\n",
"K"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4\n",
"1\n",
" \n",
"<\n",
"<\n",
"<\n",
"!\n",
"!\n",
"M\n",
"A\n",
"T\n",
"H\n",
"_\n",
"B\n",
"L\n",
"O\n",
"C\n",
"K"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4\n",
"2\n",
" \n",
"<\n",
"<\n",
"<\n",
"!\n",
"!\n",
"M\n",
"A\n",
"T\n",
"H\n",
"_\n",
"B\n",
"L\n",
"O\n",
"C\n",
"K"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\gamma_{xy} = \\sum_{i=0}^{1}\\frac{y_ix_{i}}{\\sigma_i^2},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and show that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4\n",
"4\n",
" \n",
"<\n",
"<\n",
"<\n",
"!\n",
"!\n",
"M\n",
"A\n",
"T\n",
"H\n",
"_\n",
"B\n",
"L\n",
"O\n",
"C\n",
"K"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients $\\beta_i$. A better approach is to use the Singular Value Decomposition (SVD) method discussed below.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## The singular value decompostion\n",
"How can we use the singular value decomposition to find the parameters $\\beta_j$? More details will come. We first note that a general $m\\times n$ matrix $\\hat{A}$ can be written in terms of a diagonal matrix $\\hat{\\Sigma}$ of dimensionality $n\\times n$ and two orthognal matrices $\\hat{U}$ and $\\hat{V}$, where the first has dimensionality $m \\times n$ and the last dimensionality $n\\times n$. We have then"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{A} = \\hat{U}\\hat{\\Sigma}\\hat{V}\n",
"$$"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue View Git Blame Copy Permalink