lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

more on steepest descent

Browse Source
This commit is contained in:
mhjensen
2018-09-27 04:47:50 +02:00
parent 20e8b8fbec
commit a55882e5a4
45 changed files with 2914 additions and 4593 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/Splines/Splines.do.txt
+99 -255
View File
@@ -405,7 +405,7 @@ where $\hat{r}$ is the so-called residual or error in the iterative process.
When we have found the exact solution, $\hat{r}=0$.
!split
===== Conjugate gradient method =====
===== Gradient method =====
The residual is zero when we reach the minimum of the quadratic equation
!bt
@@ -420,7 +420,104 @@ variance, then the matrix $\hat{A}$, which is called the Hessian, is
given by the second-derivative of the function we want to minimize.
This quantity is always positive definite.
More details will be added here soon.
!split
===== Steepest descent method =====
We denote the initial guess for $\hat{x}$ as $\hat{x}_0$.
We can assume without loss of generality that
!bt
\begin{equation*}
\hat{x}_0=0,
\end{equation*}
!et
or consider the system
!bt
\begin{equation*}
\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0,
\end{equation*}
!et
instead.
!split
===== Steepest descent method =====
!bblock
One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form
!bt
\begin{equation*}
f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n.
\end{equation*}
!et
This suggests taking the first basis vector $\hat{p}_1$
to be the gradient of $f$ at $\hat{x}=\hat{x}_0$,
which equals
!bt
\begin{equation*}
\hat{A}\hat{x}_0-\hat{b},
\end{equation*}
!et
and
$\hat{x}_0=0$ it is equal $-\hat{b}$.
!eblock
!split
===== Gradient descent method =====
!bblock
Let $\hat{r}_k$ be the residual at the $k$-th step:
!bt
\begin{equation*}
\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k.
\end{equation*}
!et
Note that $\hat{r}_k$ is the negative gradient of $f$ at
$\hat{x}=\hat{x}_k$,
so the gradient descent method would be to move in the direction $\hat{r}_k$.
This gives the following expression
!bt
\begin{equation*}
\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k.
\end{equation*}
!et
!eblock
!split
===== Final expressions =====
!bblock
We can also compute the residual iteratively as
!bt
\begin{equation*}
\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1},
\end{equation*}
!et
which equals
!bt
\begin{equation*}
\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k),
\end{equation*}
!et
or
!bt
\begin{equation*}
(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k,
\end{equation*}
!et
which gives
!bt
\begin{equation*}
\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k},
\end{equation*}
!et
!eblock
!split
===== The Steepest descent algorithm =====
!split
===== Simple codes for steepest descent and conjugate gradient using a $2\times 2$ matrix, in c++, Python code to come =====
@@ -446,11 +543,7 @@ int main(int argc, char * argv[]){
cout << "The Matrix A that we are using: " << endl;
A.Print();
cout << endl;
x = ConjugateGradient(A,b,x0);
xsd = SteepestDescent(A,b,x0);
cout << "The approximate solution using Conjugate Gradient is: " << endl;
x.Print();
cout << endl;
cout << "The approximate solution using Steepest Descent is: " << endl;
xsd.Print();
cout << endl;
@@ -895,255 +988,6 @@ print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
!split
===== Conjugate gradient (CG) method =====
!bblock
The success of the CG method for finding solutions of non-linear problems is based
on the theory of conjugate gradients for linear systems of equations. It belongs
to the class of iterative methods for solving problems from linear algebra of the type
!bt
\begin{equation*}
\hat{A}\hat{x} = \hat{b}.
\end{equation*}
!et
In the iterative process we end up with a problem like
!bt
\begin{equation*}
\hat{r}= \hat{b}-\hat{A}\hat{x},
\end{equation*}
!et
where $\hat{r}$ is the so-called residual or error in the iterative process.
When we have found the exact solution, $\hat{r}=0$.
!eblock
!split
===== Conjugate gradient method =====
!bblock
The residual is zero when we reach the minimum of the quadratic equation
!bt
\begin{equation*}
P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b},
\end{equation*}
!et
with the constraint that the matrix $\hat{A}$ is positive definite and symmetric.
If we search for a minimum of the quantum mechanical variance, then the matrix
$\hat{A}$, which is called the Hessian, is given by the second-derivative of the function we want to minimize. This quantity is always positive definite. In our case this corresponds normally to the second derivative of the energy.
!eblock
!split
===== Conjugate gradient method, Newton's method first =====
!bblock
We seek the minimum of the energy or the variance as function of various variational parameters.
In our case we have thus a function $f$ whose minimum we are seeking.
In Newton's method we set $\nabla f = 0$ and we can thus compute the next iteration point
!bt
\begin{equation*}
\hat{x}-\hat{x}_i=\hat{A}^{-1}\nabla f(\hat{x}_i).
\end{equation*}
!et
Subtracting this equation from that of $\hat{x}_{i+1}$ we have
!bt
\begin{equation*}
\hat{x}_{i+1}-\hat{x}_i=\hat{A}^{-1}(\nabla f(\hat{x}_{i+1})-\nabla f(\hat{x}_i)).
\end{equation*}
!et
!eblock
!split
===== Conjugate gradient method =====
!bblock
In the CG method we define so-called conjugate directions and two vectors
$\hat{s}$ and $\hat{t}$
are said to be
conjugate if
!bt
\begin{equation*}
\hat{s}^T\hat{A}\hat{t}= 0.
\end{equation*}
!et
The philosophy of the CG method is to perform searches in various conjugate directions
of our vectors $\hat{x}_i$ obeying the above criterion, namely
!bt
\begin{equation*}
\hat{x}_i^T\hat{A}\hat{x}_j= 0.
\end{equation*}
!et
Two vectors are conjugate if they are orthogonal with respect to
this inner product. Being conjugate is a symmetric relation: if $\hat{s}$ is conjugate to $\hat{t}$, then $\hat{t}$ is conjugate to $\hat{s}$.
!eblock
!split
===== Conjugate gradient method =====
!bblock
An example is given by the eigenvectors of the matrix
!bt
\begin{equation*}
\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j,
\end{equation*}
!et
which is zero unless $i=j$.
!eblock
!split
===== Conjugate gradient method =====
!bblock
Assume now that we have a symmetric positive-definite matrix $\hat{A}$ of size
$n\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector
!bt
\begin{equation*}
\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}.
\end{equation*}
!et
We assume that $\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions.
Then the $\hat{p}_{i}$ form a basis of $R^n$ and we can expand the solution
$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely
!bt
\begin{equation*}
\hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i.
\end{equation*}
!et
!eblock
!split
===== Conjugate gradient method =====
!bblock
The coefficients are given by
!bt
\begin{equation*}
\mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}.
\end{equation*}
!et
Multiplying with $\hat{p}_k^T$ from the left gives
!bt
\begin{equation*}
\hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b},
\end{equation*}
!et
and we can define the coefficients $\alpha_k$ as
!bt
\begin{equation*}
\alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k}
\end{equation*}
!et
!eblock
!split
===== Conjugate gradient method and iterations =====
!bblock
If we choose the conjugate vectors $\hat{p}_k$ carefully,
then we may not need all of them to obtain a good approximation to the solution
$\hat{x}$.
We want to regard the conjugate gradient method as an iterative method.
This will us to solve systems where $n$ is so large that the direct
method would take too much time.
We denote the initial guess for $\hat{x}$ as $\hat{x}_0$.
We can assume without loss of generality that
!bt
\begin{equation*}
\hat{x}_0=0,
\end{equation*}
!et
or consider the system
!bt
\begin{equation*}
\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0,
\end{equation*}
!et
instead.
!eblock
!split
===== Conjugate gradient method =====
!bblock
One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form
!bt
\begin{equation*}
f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n.
\end{equation*}
!et
This suggests taking the first basis vector $\hat{p}_1$
to be the gradient of $f$ at $\hat{x}=\hat{x}_0$,
which equals
!bt
\begin{equation*}
\hat{A}\hat{x}_0-\hat{b},
\end{equation*}
!et
and
$\hat{x}_0=0$ it is equal $-\hat{b}$.
The other vectors in the basis will be conjugate to the gradient,
hence the name conjugate gradient method.
!eblock
!split
===== Conjugate gradient method =====
!bblock
Let $\hat{r}_k$ be the residual at the $k$-th step:
!bt
\begin{equation*}
\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k.
\end{equation*}
!et
Note that $\hat{r}_k$ is the negative gradient of $f$ at
$\hat{x}=\hat{x}_k$,
so the gradient descent method would be to move in the direction $\hat{r}_k$.
Here, we insist that the directions $\hat{p}_k$ are conjugate to each other,
so we take the direction closest to the gradient $\hat{r}_k$
under the conjugacy constraint.
This gives the following expression
!bt
\begin{equation*}
\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k.
\end{equation*}
!et
!eblock
!split
===== Conjugate gradient method =====
!bblock
We can also compute the residual iteratively as
!bt
\begin{equation*}
\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1},
\end{equation*}
!et
which equals
!bt
\begin{equation*}
\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k),
\end{equation*}
!et
or
!bt
\begin{equation*}
(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k,
\end{equation*}
!et
which gives
!bt
\begin{equation*}
\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k},
\end{equation*}
!et
!eblock