{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- dom:TITLE: Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis -->\n",
    "# Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis\n",
    "<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
    "<!-- Author: -->  \n",
    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
    "\n",
    "Date: **Nov 21, 2017**\n",
    "\n",
    "Copyright 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Regression analysis, overarching aims\n",
    "\n",
    "Regression modeling deals with the description of  the sampling distribution of a given random variable $y$ varies as function of another variable or a set of such variables $\\hat{x} =[x_0, x_1,\\dots, x_p]^T$. \n",
    "The first variable is called the **dependent**, the **outcome** or the **response** variable while the set of variables $\\hat{x}$ is called the independent variable, or the predictor variable or the explanatory variable. \n",
    "\n",
    "A regression model aims at finding a likelihood function $p(y\\vert \\hat{x})$, that is the conditional distribution for $y$ with a given $\\hat{x}$. The estimation of  $p(y\\vert \\hat{x})$ is made using a data set with \n",
    "* $n$ cases $i = 0, 1, 2, \\dots, n-1$ \n",
    "\n",
    "* Response (dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n",
    "\n",
    "* $p$ Explanatory (independent or predictor) variables $\\hat{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip}]$ with $i = 0, 1, 2, \\dots, n-1$   \n",
    "\n",
    " The goal of the regression analysis is to extract/exploit relationship between $y_i$ and $\\hat{x}_i$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions .\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## General linear models\n",
    "Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\\hat{y}=[y_0,y_1,\\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\\hat{x}=[x_0,x_1,\\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. \n",
    "\n",
    "Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_i x_i^j+\\epsilon_i,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $\\epsilon_i$ is the error in our approximation.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Rewriting the fitting procedure as a linear algebra problem\n",
    "For every set of values $y_i,x_i$ we have thus the corresponding set of equations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "y_0&=\\beta_0+\\beta_1x_0^1+\\beta_2x_0^2+\\dots+\\beta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n",
    "y_1&=\\beta_0+\\beta_1x_1^1+\\beta_2x_1^2+\\dots+\\beta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n",
    "y_2&=\\beta_0+\\beta_1x_2^1+\\beta_2x_2^2+\\dots+\\beta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n",
    "\\dots & \\dots \\\\\n",
    "y_{n-1}&=\\beta_0+\\beta_1x_{n-1}^1+\\beta_2x_{n-1}^2+\\dots+\\beta_1x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rewriting the fitting procedure as a linear algebra problem, follows\n",
    "Defining the vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2\n",
    " \n",
    "<\n",
    "<\n",
    "<\n",
    "!\n",
    "!\n",
    "M\n",
    "A\n",
    "T\n",
    "H\n",
    "_\n",
    "B\n",
    "L\n",
    "O\n",
    "C\n",
    "K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3\n",
    " \n",
    "<\n",
    "<\n",
    "<\n",
    "!\n",
    "!\n",
    "M\n",
    "A\n",
    "T\n",
    "H\n",
    "_\n",
    "B\n",
    "L\n",
    "O\n",
    "C\n",
    "K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{\\epsilon} = [\\epsilon_0,\\epsilon_1, \\epsilon_2,\\dots, \\epsilon_{n-1}]^T,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{X}=\n",
    "\\begin{bmatrix} \n",
    "1& x_{0}^1 &x_{0}^2& \\dots & \\dots &x_{0}^{n-1}\\\\\n",
    "1& x_{1}^1 &x_{1}^2& \\dots & \\dots &x_{1}^{n-1}\\\\\n",
    "1& x_{2}^1 &x_{2}^2& \\dots & \\dots &x_{2}^{n-1}\\\\                      \n",
    "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
    "1& x_{n-1}^1 &x_{n-1}^2& \\dots & \\dots &x_{n-1}^{n-1}\\\\\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can rewrite our equations as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{y} = \\hat{X}\\hat{\\beta}+\\hat{\\epsilon}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generalizing the fitting procedure as a linear algebra problem\n",
    "We are obviously not limited to the above polynomial. We could replace the various powers of $x$ with elements of Fourier series, that is, instead of $x_i^j$ we could have $\\cos{(j x_i)}$ or $\\sin{(j x_i)}$, or time series or other orthogonal functions.\n",
    "For every set of values $y_i,x_i$ we can then generalize the equations to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n",
    "\\dots & \\dots \\\\\n",
    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_i\\\\\n",
    "\\dots & \\dots \\\\\n",
    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_1x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generalizing the fitting procedure as a linear algebra problem\n",
    "We redefine in turn the matrix $\\hat{X}$ as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{X}=\n",
    "\\begin{bmatrix} \n",
    "x_{00}& x_{01} &x_{02}& \\dots & \\dots &x_{0,n-1}\\\\\n",
    "x_{10}& x_{11} &x_{12}& \\dots & \\dots &x_{1,n-1}\\\\\n",
    "x_{20}& x_{21} &x_{22}& \\dots & \\dots &x_{2,n-1}\\\\                      \n",
    "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
    "x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,n-1}\\\\\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and without loss of generality we rewrite again  our equations as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{y} = \\hat{X}\\hat{\\beta}+\\hat{\\epsilon}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The left-hand side of this equation forms know. Our error vector $\\hat{\\epsilon}$ and the parameter vector $\\hat{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values?\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Optimizing our parameters\n",
    "We have defined the matrix $\\hat{X}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align*}\n",
    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n",
    "\\dots & \\dots \\\\\n",
    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_1\\\\\n",
    "\\dots & \\dots \\\\\n",
    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_1x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizing our parameters, more details\n",
    "We well use this matrix to define the approximation $\\hat{\\tilde{y}}$ via the unknown quantity $\\hat{\\beta}$ as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{\\tilde{y}}= \\hat{X}\\hat{\\beta},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parametrized values $\\tilde{y}_i$, namely"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "Q(\\hat{\\beta})=\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\left(\\hat{y}-\\hat{\\tilde{y}}\\right)^T\\left(\\hat{y}-\\hat{\\tilde{y}}\\right),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or using the matrix $\\hat{X}$ as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "Q(\\hat{\\beta})=\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right)^T\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretations and optimizing our parameters\n",
    "The function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "Q(\\hat{\\beta})=\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right)^T\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value of for example a numerical  experiment. When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that till now  we have treated $y_i$ as the exact value. Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable.\n",
    "\n",
    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $Q(\\hat{\\beta})$ by requiring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which results in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\beta_j} = -2\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or in a matrix-vector form as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretations and optimizing our parameters\n",
    "We can rewrite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{X}^T\\hat{y} = \\hat{X}^T\\hat{X}\\hat{\\beta},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and if the matrix $\\hat{X}^T\\hat{X}$ is invertible we have the solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{\\beta} =\\left(\\hat{X}^T\\hat{X}\\right)^{-1}\\hat{X}^T\\hat{y}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretations and optimizing our parameters\n",
    "The residuals $\\hat{\\epsilon}$ are in turn given by"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{\\epsilon} = \\hat{y}-\\hat{\\tilde{y}} = \\hat{y}-\\hat{X}\\hat{\\beta},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and with"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right)= 0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{X}^T\\hat{\\epsilon}=\\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right)= 0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "meaning that the solution for $\\hat{\\beta}$ is the one which minimizes the residuals.  Later we will link this with the maximum likelihood approach.\n",
    "\n",
    "\n",
    "\n",
    "## The $\\chi^2$ function\n",
    "\n",
    "Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable.\n",
    "\n",
    "Introducing the standard deviation $\\sigma_i$ for each measurement $y_i$, we define now the $\\chi^2$ function as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\chi^2(\\hat{\\beta})=\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\left(\\hat{y}-\\hat{\\tilde{y}}\\right)^T\\frac{1}{\\hat{\\Sigma^2}}\\left(\\hat{y}-\\hat{\\tilde{y}}\\right),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where the matrix $\\hat{\\Sigma}$ is a diagonal matrix with $\\sigma_i$ as matrix elements.\n",
    "\n",
    "\n",
    "\n",
    "## The $\\chi^2$ function\n",
    "\n",
    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $\\chi^2(\\hat{\\beta})$ by requiring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which results in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_j} = -2\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or in a matrix-vector form as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{A}^T\\left( \\hat{b}-\\hat{A}\\hat{\\beta}\\right).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where we have defined the matrix $\\hat{A} =\\hat{X}/\\hat{\\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\\sigma_i$ and the vector $\\hat{b}$ with elements $b_i = y_i/\\sigma_i$.\n",
    "\n",
    "\n",
    "\n",
    "## The $\\chi^2$ function\n",
    "\n",
    "We can rewrite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{A}^T\\left( \\hat{b}-\\hat{A}\\hat{\\beta}\\right),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{A}^T\\hat{b} = \\hat{A}^T\\hat{A}\\hat{\\beta},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and if the matrix $\\hat{A}^T\\hat{A}$ is invertible we have the solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{\\beta} =\\left(\\hat{A}^T\\hat{A}\\right)^{-1}\\hat{A}^T\\hat{b}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The $\\chi^2$ function\n",
    "\n",
    "If we then introduce the matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{H} =  \\hat{A}^T\\hat{A},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we have then the following expression for the parameters $\\beta_j$ (the matrix elements of $\\hat{H}$ are $h_{ij}$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\beta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We state without proof the expression for the uncertainty  in the parameters $\\beta_j$ as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sigma^2(\\beta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\beta_j}{\\partial y_i}\\right)^2,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "resulting in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sigma^2(\\beta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The $\\chi^2$ function\n",
    "The first step here is to approximate the function $y$ with a first-order polynomial, that is we write"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "y=y(x) \\rightarrow y(x_i) \\approx \\beta_0+\\beta_1 x_i.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By computing the derivatives of $\\chi^2$ with respect to $\\beta_0$ and $\\beta_1$ show that these are given by"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_0} = -2\\left[ \\sum_{i=0}^{1}\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_0} = -2\\left[ \\sum_{i=0}^{1}x_i\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The $\\chi^2$ function\n",
    "\n",
    "We define then"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\gamma =  \\sum_{i=0}^{1}\\frac{1}{\\sigma_i^2},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4\n",
    "0\n",
    " \n",
    "<\n",
    "<\n",
    "<\n",
    "!\n",
    "!\n",
    "M\n",
    "A\n",
    "T\n",
    "H\n",
    "_\n",
    "B\n",
    "L\n",
    "O\n",
    "C\n",
    "K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4\n",
    "1\n",
    " \n",
    "<\n",
    "<\n",
    "<\n",
    "!\n",
    "!\n",
    "M\n",
    "A\n",
    "T\n",
    "H\n",
    "_\n",
    "B\n",
    "L\n",
    "O\n",
    "C\n",
    "K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4\n",
    "2\n",
    " \n",
    "<\n",
    "<\n",
    "<\n",
    "!\n",
    "!\n",
    "M\n",
    "A\n",
    "T\n",
    "H\n",
    "_\n",
    "B\n",
    "L\n",
    "O\n",
    "C\n",
    "K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\gamma_{xy} = \\sum_{i=0}^{1}\\frac{y_ix_{i}}{\\sigma_i^2},\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and show that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4\n",
    "4\n",
    " \n",
    "<\n",
    "<\n",
    "<\n",
    "!\n",
    "!\n",
    "M\n",
    "A\n",
    "T\n",
    "H\n",
    "_\n",
    "B\n",
    "L\n",
    "O\n",
    "C\n",
    "K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\beta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients $\\beta_i$.  A better approach is to use the Singular Value Decomposition (SVD) method discussed below.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## The singular value decompostion\n",
    "How can we use the singular value decomposition to find the parameters $\\beta_j$? More details will come. We first note that a general $m\\times n$ matrix $\\hat{A}$ can be written in terms of a diagonal matrix $\\hat{\\Sigma}$ of dimensionality $n\\times n$ and two orthognal matrices $\\hat{U}$ and $\\hat{V}$, where the first has dimensionality $m \\times n$ and the last dimensionality $n\\times n$. We have then"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\hat{A} = \\hat{U}\\hat{\\Sigma}\\hat{V}\n",
    "$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}