lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
930ffb34fa39c5bb6bae5283f1c3dc4f2ebf4e6c
FYS-STK4155/doc/pub/Splines/ipynb/Splines.ipynb
T
mhjensen 4e5fa98fe8 General update of several files with to do list
2018-05-22 16:49:43 -04:00

1048 lines
26 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- dom:TITLE: Data Analysis and Machine Learning Lectures: Cubic Splines and Gradient Methods -->\n",
"# Data Analysis and Machine Learning Lectures: Cubic Splines and Gradient Methods\n",
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **May 22, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"\n",
"## Cubic Splines\n",
"Cubic spline interpolation is among one of the most used \n",
"methods for interpolating between data points where the arguments\n",
"are organized as ascending series. In the library program we supply\n",
"such a function, based on the so-called cubic spline method to be \n",
"described below. \n",
"\n",
"A spline function consists of polynomial pieces defined on\n",
"subintervals. The different subintervals are connected via\n",
"various continuity relations.\n",
"\n",
"Assume we have at our disposal $n+1$ points $x_0, x_1, \\dots x_n$ \n",
"arranged so that $x_0 < x_1 < x_2 < \\dots x_{n-1} < x_n$ (such points are called\n",
"knots). A spline function $s$ of degree $k$ with $n+1$ knots is defined\n",
"as follows\n",
" * On every subinterval $[x_{i-1},x_i)$ *s* is a polynomial of degree $\\le k$.\n",
"\n",
" * $s$ has $k-1$ continuous derivatives in the whole interval $[x_0,x_n]$.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Splines\n",
"As an example, consider a spline function of degree $k=1$ defined as follows"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s(x)=\\begin{bmatrix} s_0(x)=a_0x+b_0 & x\\in [x_0, x_1) \\\\ \n",
" s_1(x)=a_1x+b_1 & x\\in [x_1, x_2) \\\\ \n",
" \\dots & \\dots \\\\\n",
" s_{n-1}(x)=a_{n-1}x+b_{n-1} & x\\in \n",
" [x_{n-1}, x_n] \\end{bmatrix}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this case the polynomial consists of series of straight lines \n",
"connected to each other at every endpoint. The number of continuous\n",
"derivatives is then $k-1=0$, as expected when we deal with straight lines.\n",
"Such a polynomial is quite easy to construct given\n",
"$n+1$ points $x_0, x_1, \\dots x_n$ and their corresponding \n",
"function values.\n",
"\n",
"\n",
"\n",
"\n",
"## Splines\n",
"The most commonly used spline function is the one with $k=3$, the so-called\n",
"cubic spline function. \n",
"Assume that we have in adddition to the $n+1$ knots a series of\n",
"functions values $y_0=f(x_0), y_1=f(x_1), \\dots y_n=f(x_n)$.\n",
"By definition, the polynomials $s_{i-1}$ and $s_i$ \n",
"are thence supposed to interpolate the same point $i$, that is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s_{i-1}(x_i)= y_i = s_i(x_i),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"with $1 \\le i \\le n-1$. In total we have $n$ polynomials of the \n",
"type"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s_i(x)=a_{i0}+a_{i1}x+a_{i2}x^2+a_{i2}x^3,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"yielding $4n$ coefficients to determine.\n",
"\n",
"\n",
"\n",
"\n",
"## Splines\n",
"Every subinterval provides in addition the $2n$ conditions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y_i = s(x_i),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s(x_{i+1})= y_{i+1},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"to be fulfilled. If we also assume that $s'$ and $s''$ are continuous,\n",
"then"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s'_{i-1}(x_i)= s'_i(x_i),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"yields $n-1$ conditions. Similarly,"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s''_{i-1}(x_i)= s''_i(x_i),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"results in additional $n-1$ conditions. In total we have $4n$ coefficients\n",
"and $4n-2$ equations to determine them, leaving us with $2$ degrees of \n",
"freedom to be determined.\n",
"\n",
"\n",
"\n",
"## Splines\n",
"Using the last equation we define two values for the second derivative, namely"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s''_{i}(x_i)= f_i,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s''_{i}(x_{i+1})= f_{i+1},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and setting up a straight line between $f_i$ and $f_{i+1}$ we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s_i''(x) = \\frac{f_i}{x_{i+1}-x_i}(x_{i+1}-x)+\n",
" \\frac{f_{i+1}}{x_{i+1}-x_i}(x-x_i),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and integrating twice one obtains"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s_i(x) = \\frac{f_i}{6(x_{i+1}-x_i)}(x_{i+1}-x)^3+\n",
" \\frac{f_{i+1}}{6(x_{i+1}-x_i)}(x-x_i)^3\n",
" +c(x-x_i)+d(x_{i+1}-x).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Splines\n",
"Using the conditions $s_i(x_i)=y_i$ and $s_i(x_{i+1})=y_{i+1}$ \n",
"we can in turn determine the constants $c$ and $d$ resulting in"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s_i(x) =\\frac{f_i}{6(x_{i+1}-x_i)}(x_{i+1}-x)^3+\n",
" \\frac{f_{i+1}}{6(x_{i+1}-x_i)}(x-x_i)^3 \\nonumber\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"_auto1\"></div>\n",
"\n",
"$$\n",
"\\begin{equation} \n",
" +(\\frac{y_{i+1}}{x_{i+1}-x_i}-\\frac{f_{i+1}(x_{i+1}-x_i)}{6})\n",
" (x-x_i)+\n",
" (\\frac{y_{i}}{x_{i+1}-x_i}-\\frac{f_{i}(x_{i+1}-x_i)}{6})\n",
" (x_{i+1}-x).\n",
"\\label{_auto1} \\tag{1}\n",
"\\end{equation}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Splines\n",
"How to determine the values of the second\n",
"derivatives $f_{i}$ and $f_{i+1}$? We use the continuity assumption \n",
"of the first derivatives"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"s'_{i-1}(x_i)= s'_i(x_i),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and set $x=x_i$. Defining $h_i=x_{i+1}-x_i$ we obtain finally\n",
"the following expression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"h_{i-1}f_{i-1}+2(h_{i}+h_{i-1})f_i+h_if_{i+1}=\n",
" \\frac{6}{h_i}(y_{i+1}-y_i)-\\frac{6}{h_{i-1}}(y_{i}-y_{i-1}),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and introducing the shorthands $u_i=2(h_{i}+h_{i-1})$, \n",
"$v_i=\\frac{6}{h_i}(y_{i+1}-y_i)-\\frac{6}{h_{i-1}}(y_{i}-y_{i-1})$,\n",
"we can reformulate the problem as a set of linear equations to be \n",
"solved through e.g., Gaussian elemination\n",
"\n",
"\n",
"\n",
"\n",
"## Splines\n",
"Gaussian elimination"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\begin{bmatrix} u_1 & h_1 &0 &\\dots & & & & \\\\\n",
" h_1 & u_2 & h_2 &0 &\\dots & & & \\\\\n",
" 0 & h_2 & u_3 & h_3 &0 &\\dots & & \\\\\n",
" \\dots& & \\dots &\\dots &\\dots &\\dots &\\dots & \\\\\n",
" &\\dots & & &0 &h_{n-3} &u_{n-2} &h_{n-2} \\\\\n",
" & && & &0 &h_{n-2} &u_{n-1} \\end{bmatrix}\n",
" \\begin{bmatrix} f_1 \\\\ \n",
" f_2 \\\\\n",
" f_3\\\\\n",
" \\dots \\\\\n",
" f_{n-2} \\\\ \n",
" f_{n-1} \\end{bmatrix} =\n",
" \\begin{bmatrix} v_1 \\\\ \n",
" v_2 \\\\\n",
" v_3\\\\\n",
" \\dots \\\\\n",
" v_{n-2}\\\\\n",
" v_{n-1} \\end{bmatrix}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that this is a set of tridiagonal equations and can be solved \n",
"through only $O(n)$ operations.\n",
"\n",
"\n",
"\n",
"## Splines\n",
"The functions supplied in the program library are *spline* and *splint*.\n",
"In order to use cubic spline interpolation you need first to call"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" spline(double x[], double y[], int n, double yp1, double yp2, double y2[])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This function takes as\n",
"input $x[0,..,n - 1]$ and $y[0,..,n - 1]$ containing a tabulation\n",
"$y_i = f(x_i)$ with $x_0 < x_1 < .. < x_{n - 1}$ \n",
"together with the \n",
"first derivatives of $f(x)$ at $x_0$ and $x_{n-1}$, respectively. Then the\n",
"function returns $y2[0,..,n-1]$ which contains the second derivatives of\n",
"$f(x_i)$ at each point $x_i$. $n$ is the number of points.\n",
"This function provides the cubic spline interpolation for all subintervals\n",
"and is called only once.\n",
"\n",
"\n",
"\n",
"\n",
"## Splines\n",
"Thereafter, if you wish to make various interpolations, you need to call the function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" splint(double x[], double y[], double y2a[], int n, double x, double *y)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which takes as input\n",
"the tabulated values $x[0,..,n - 1]$ and $y[0,..,n - 1]$ and the output \n",
"y2a[0,..,n - 1] from *spline*. It returns the value $y$ corresponding\n",
"to the point $x$.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Conjugate gradient (CG) method\n",
"The success of the CG method for finding solutions of non-linear problems is based\n",
"on the theory of conjugate gradients for linear systems of equations. It belongs\n",
"to the class of iterative methods for solving problems from linear algebra of the type"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{A}\\hat{x} = \\hat{b}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the iterative process we end up with a problem like"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{r}= \\hat{b}-\\hat{A}\\hat{x},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\hat{r}$ is the so-called residual or error in the iterative process.\n",
"\n",
"When we have found the exact solution, $\\hat{r}=0$.\n",
"\n",
"\n",
"\n",
"\n",
"## Conjugate gradient method\n",
"\n",
"The residual is zero when we reach the minimum of the quadratic equation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"P(\\hat{x})=\\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T\\hat{b},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"with the constraint that the matrix $\\hat{A}$ is positive definite and symmetric.\n",
"If we search for a minimum of the quantum mechanical variance, then the matrix \n",
"$\\hat{A}$, which is called the Hessian, is given by the second-derivative of the function we want to minimize. This quantity is always positive definite. In our case this corresponds normally to the second derivative of the energy.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Conjugate gradient method, Newton's method first\n",
"We seek the minimum of the energy or the variance as function of various variational parameters. \n",
"In our case we have thus a function $f$ whose minimum we are seeking.\n",
"In Newton's method we set $\\nabla f = 0$ and we can thus compute the next iteration point"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{x}-\\hat{x}_i=\\hat{A}^{-1}\\nabla f(\\hat{x}_i).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Subtracting this equation from that of $\\hat{x}_{i+1}$ we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{x}_{i+1}-\\hat{x}_i=\\hat{A}^{-1}(\\nabla f(\\hat{x}_{i+1})-\\nabla f(\\hat{x}_i)).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple example and demonstration\n",
"The function $f$ can be either the energy or the variance. If we choose the energy then we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{\\alpha}_{i+1}-\\hat{\\alpha}_i=\\hat{A}^{-1}(\\nabla E(\\hat{\\alpha}_{i+1})-\\nabla E(\\hat{\\alpha}_i)).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the simple harmonic oscillator model, the gradient and the Hessian $\\hat{A}$ are"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{d\\langle E_L[\\alpha]\\rangle}{d\\alpha} = \\alpha-\\frac{1}{4\\alpha^3}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and a second derivative which is always positive (meaning that we find a minimum)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{A}= \\frac{d^2\\langle E_L[\\alpha]\\rangle}{d\\alpha^2} = 1+\\frac{3}{4\\alpha^4}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple example and demonstration\n",
"We get then"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\alpha_{i+1}=\\frac{4}{3}\\alpha_i-\\frac{\\alpha_i^4}{3\\alpha_{i+1}^3},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which can be rewritten as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\alpha_{i+1}^4-\\frac{4}{3}\\alpha_i\\alpha_{i+1}^4+\\frac{1}{3}\\alpha_i^4.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conjugate gradient method\n",
"In the CG method we define so-called conjugate directions and two vectors \n",
"$\\hat{s}$ and $\\hat{t}$\n",
"are said to be\n",
"conjugate if"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{s}^T\\hat{A}\\hat{t}= 0.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The philosophy of the CG method is to perform searches in various conjugate directions\n",
"of our vectors $\\hat{x}_i$ obeying the above criterion, namely"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{x}_i^T\\hat{A}\\hat{x}_j= 0.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Two vectors are conjugate if they are orthogonal with respect to \n",
"this inner product. Being conjugate is a symmetric relation: if $\\hat{s}$ is conjugate to $\\hat{t}$, then $\\hat{t}$ is conjugate to $\\hat{s}$.\n",
"\n",
"\n",
"\n",
"## Conjugate gradient method\n",
"An example is given by the eigenvectors of the matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{v}_i^T\\hat{A}\\hat{v}_j= \\lambda\\hat{v}_i^T\\hat{v}_j,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which is zero unless $i=j$.\n",
"\n",
"\n",
"\n",
"\n",
"## Conjugate gradient method\n",
"Assume now that we have a symmetric positive-definite matrix $\\hat{A}$ of size\n",
"$n\\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{x}_{i+1}=\\hat{x}_{i}+\\alpha_i\\hat{p}_{i}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We assume that $\\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n",
"Then the $\\hat{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n",
"$ \\hat{A}\\hat{x} = \\hat{b}$ in this basis, namely"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{x} = \\sum^{n}_{i=1} \\alpha_i \\hat{p}_i.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conjugate gradient method\n",
"The coefficients are given by"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Multiplying with $\\hat{p}_k^T$ from the left gives"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{p}_k^T \\hat{A}\\hat{x} = \\sum^{n}_{i=1} \\alpha_i\\hat{p}_k^T \\hat{A}\\hat{p}_i= \\hat{p}_k^T \\hat{b},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and we can define the coefficients $\\alpha_k$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\alpha_k = \\frac{\\hat{p}_k^T \\hat{b}}{\\hat{p}_k^T \\hat{A} \\hat{p}_k}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conjugate gradient method and iterations\n",
"\n",
"If we choose the conjugate vectors $\\hat{p}_k$ carefully, \n",
"then we may not need all of them to obtain a good approximation to the solution \n",
"$\\hat{x}$. \n",
"We want to regard the conjugate gradient method as an iterative method. \n",
"This will us to solve systems where $n$ is so large that the direct \n",
"method would take too much time.\n",
"\n",
"We denote the initial guess for $\\hat{x}$ as $\\hat{x}_0$. \n",
"We can assume without loss of generality that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{x}_0=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or consider the system"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{A}\\hat{z} = \\hat{b}-\\hat{A}\\hat{x}_0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"instead.\n",
"\n",
"\n",
"\n",
"\n",
"## Conjugate gradient method\n",
"One can show that the solution $\\hat{x}$ is also the unique minimizer of the quadratic form"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"f(\\hat{x}) = \\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T \\hat{x} , \\quad \\hat{x}\\in\\mathbf{R}^n.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This suggests taking the first basis vector $\\hat{p}_1$ \n",
"to be the gradient of $f$ at $\\hat{x}=\\hat{x}_0$, \n",
"which equals"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{A}\\hat{x}_0-\\hat{b},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and \n",
"$\\hat{x}_0=0$ it is equal $-\\hat{b}$.\n",
"The other vectors in the basis will be conjugate to the gradient, \n",
"hence the name conjugate gradient method.\n",
"\n",
"\n",
"\n",
"\n",
"## Conjugate gradient method\n",
"Let $\\hat{r}_k$ be the residual at the $k$-th step:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{r}_k=\\hat{b}-\\hat{A}\\hat{x}_k.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that $\\hat{r}_k$ is the negative gradient of $f$ at \n",
"$\\hat{x}=\\hat{x}_k$, \n",
"so the gradient descent method would be to move in the direction $\\hat{r}_k$. \n",
"Here, we insist that the directions $\\hat{p}_k$ are conjugate to each other, \n",
"so we take the direction closest to the gradient $\\hat{r}_k$ \n",
"under the conjugacy constraint. \n",
"This gives the following expression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{p}_{k+1}=\\hat{r}_k-\\frac{\\hat{p}_k^T \\hat{A}\\hat{r}_k}{\\hat{p}_k^T\\hat{A}\\hat{p}_k} \\hat{p}_k.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conjugate gradient method\n",
"We can also compute the residual iteratively as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{r}_{k+1}=\\hat{b}-\\hat{A}\\hat{x}_{k+1},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which equals"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{b}-\\hat{A}(\\hat{x}_k+\\alpha_k\\hat{p}_k),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"(\\hat{b}-\\hat{A}\\hat{x}_k)-\\alpha_k\\hat{A}\\hat{p}_k,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which gives"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{r}_{k+1}=\\hat{r}_k-\\hat{A}\\hat{p}_{k},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Gradient Descent codes"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"\n",
"# Importing various packages\n",
"from math import exp, sqrt\n",
"from random import random, seed\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from mpl_toolkits.mplot3d import Axes3D\n",
"from matplotlib import cm\n",
"from matplotlib.ticker import LinearLocator, FormatStrFormatter\n",
"import sys\n",
"\n",
"x = 2*np.random.rand(100,1)\n",
"y = 4+3*x+np.random.randn(100,1)\n",
"\n",
"xb = np.c_[np.ones((100,1)), x]\n",
"theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
"print(theta_linreg)\n",
"theta = np.random.randn(2,1)\n",
"\n",
"eta = 0.1\n",
"Niterations = 1000\n",
"m = 100\n",
"\n",
"for iter in range(Niterations):\n",
" gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)\n",
" theta -= eta*gradients\n",
"\n",
"print(theta)\n",
"xnew = np.array([[0],[2]])\n",
"xbnew = np.c_[np.ones((2,1)), xnew]\n",
"ypredict = xbnew.dot(theta)\n",
"ypredict2 = xbnew.dot(theta_linreg)\n",
"plt.plot(xnew, ypredict, \"r-\")\n",
"plt.plot(xnew, ypredict2, \"b-\")\n",
"plt.plot(x, y ,'ro')\n",
"plt.axis([0,2.0,0, 15.0])\n",
"plt.xlabel(r'$x$')\n",
"plt.ylabel(r'$y$')\n",
"plt.title(r'Random numbers ')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Importing various packages\n",
"from math import exp, sqrt\n",
"from random import random, seed\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.linear_model import SGDRegressor\n",
"\n",
"x = 2*np.random.rand(100,1)\n",
"y = 4+3*x+np.random.randn(100,1)\n",
"\n",
"xb = np.c_[np.ones((100,1)), x]\n",
"theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
"print(theta_linreg)\n",
"sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n",
"sgdreg.fit(x,y.ravel())\n",
"print(sgdreg.intercept_, sgdreg.coef_)"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue View Git Blame Copy Permalink