1
print(accuracy)
print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-
import scikitplot as skplt
y_pred = logreg.predict(X_test)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
@@ -2041,529 +1997,6 @@ is minimal, where \( f \) is convex and differentiable. Then, any point
Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
-
-Standard steepest descent
-
-Before we proceed, we would like to discuss the approach called the
-standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able
-to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).
-
-
-The success of the CG method
-for finding solutions of non-linear problems is based on the theory
-of conjugate gradients for linear systems of equations. It belongs to
-the class of iterative methods for solving problems from linear
-algebra of the type
-
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}.
-\end{equation*}
-$$
-
-In the iterative process we end up with a problem like
-
-$$
-\begin{equation*}
- \boldsymbol{r}= \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x},
-\end{equation*}
-$$
-
-where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.
-
-When we have found the exact solution, \( \boldsymbol{r}=0 \).
-
-
-Gradient method
-
-The residual is zero when we reach the minimum of the quadratic equation
-$$
-\begin{equation*}
- P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b},
-\end{equation*}
-$$
-
-with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and
-symmetric. This defines also the Hessian and we want it to be positive definite.
-
-
-
-Steepest descent method
-
-We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \).
-We can assume without loss of generality that
-
-$$
-\begin{equation*}
-\boldsymbol{x}_0=0,
-\end{equation*}
-$$
-
-or consider the system
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0,
-\end{equation*}
-$$
-
-instead.
-
-
-Steepest descent method
-
-
-
-
One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form
-$$
-\begin{equation*}
- f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n.
-\end{equation*}
-$$
-
-
This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition)
-to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \),
-which equals
-
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b},
-\end{equation*}
-$$
-
-
and
-\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).
-
-
-Final expressions
-
-
-
-
We can compute the residual iteratively as
-$$
-\begin{equation*}
-\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1},
- \end{equation*}
-$$
-
-
which equals
-$$
-\begin{equation*}
-\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k),
- \end{equation*}
-$$
-
-
or
-$$
-\begin{equation*}
-(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k,
- \end{equation*}
-$$
-
-
which gives
-
-$$
-\alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k}
-$$
-
-
leading to the iterative scheme
-$$
-\begin{equation*}
-\boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k},
- \end{equation*}
-$$
-
-Steepest descent example
-
-
-
-
-
-And then as countor plot
-
-
-
-
-Find guesses
-
-
-
-
-Run it!
-
-
-
-
-What happened?
-
-
-
-
-Note that we did only one iteration here. We can easily add more using our previous guesses.
-
-
-Conjugate gradient method
-
-
-
-
In the CG method we define so-called conjugate directions and two vectors
-\( \boldsymbol{s} \) and \( \boldsymbol{t} \)
-are said to be
-conjugate if
-
-$$
-\begin{equation*}
-\boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0.
-\end{equation*}
-$$
-
-
The philosophy of the CG method is to perform searches in various conjugate directions
-of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely
-
-$$
-\begin{equation*}
-\boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0.
-\end{equation*}
-$$
-
-
Two vectors are conjugate if they are orthogonal with respect to
-this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \).
-
-
-Conjugate gradient method
-
-
-
-
An example is given by the eigenvectors of the matrix
-$$
-\begin{equation*}
-\boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j,
-\end{equation*}
-$$
-
-
which is zero unless \( i=j \).
-
-Conjugate gradient method
-
-
-
-
Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size
-\( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector
-
-$$
-\begin{equation*}
-\boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}.
-\end{equation*}
-$$
-
-
We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions.
-Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution
-$ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely
-
-
-$$
-\begin{equation*}
- \boldsymbol{x} = \sum^{n}_{i=1} \alpha_i \boldsymbol{p}_i.
-\end{equation*}
-$$
-
-Conjugate gradient method
-
-
-
-
The coefficients are given by
-$$
-\begin{equation*}
- \mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}.
-\end{equation*}
-$$
-
-
Multiplying with \( \boldsymbol{p}_k^T \) from the left gives
-
-$$
-\begin{equation*}
- \boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{x} = \sum^{n}_{i=1} \alpha_i\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{p}_i= \boldsymbol{p}_k^T \boldsymbol{b},
-\end{equation*}
-$$
-
-
and we can define the coefficients \( \alpha_k \) as
-
-$$
-\begin{equation*}
- \alpha_k = \frac{\boldsymbol{p}_k^T \boldsymbol{b}}{\boldsymbol{p}_k^T \boldsymbol{A} \boldsymbol{p}_k}
-\end{equation*}
-$$
-
-Conjugate gradient method and iterations
-
-
-
-
-
If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully,
-then we may not need all of them to obtain a good approximation to the solution
-\( \boldsymbol{x} \).
-We want to regard the conjugate gradient method as an iterative method.
-This will us to solve systems where \( n \) is so large that the direct
-method would take too much time.
-
-
-
We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \).
-We can assume without loss of generality that
-
-$$
-\begin{equation*}
-\boldsymbol{x}_0=0,
-\end{equation*}
-$$
-
-
or consider the system
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0,
-\end{equation*}
-$$
-
-
instead.
-
-Conjugate gradient method
-
-
-
-
One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form
-$$
-\begin{equation*}
- f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n.
-\end{equation*}
-$$
-
-
This suggests taking the first basis vector \( \boldsymbol{p}_1 \)
-to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \),
-which equals
-
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b},
-\end{equation*}
-$$
-
-
and
-\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).
-The other vectors in the basis will be conjugate to the gradient,
-hence the name conjugate gradient method.
-
-
-Conjugate gradient method
-
-
-
-
Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step:
-$$
-\begin{equation*}
-\boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k.
-\end{equation*}
-$$
-
-
Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at
-\( \boldsymbol{x}=\boldsymbol{x}_k \),
-so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \).
-Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other,
-so we take the direction closest to the gradient \( \boldsymbol{r}_k \)
-under the conjugacy constraint.
-This gives the following expression
-
-$$
-\begin{equation*}
-\boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k.
-\end{equation*}
-$$
-
-Conjugate gradient method
-
-
-
-
We can also compute the residual iteratively as
-$$
-\begin{equation*}
-\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1},
- \end{equation*}
-$$
-
-
which equals
-$$
-\begin{equation*}
-\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k),
- \end{equation*}
-$$
-
-
or
-$$
-\begin{equation*}
-(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k,
- \end{equation*}
-$$
-
-
which gives
-
-$$
-\begin{equation*}
-\boldsymbol{r}_{k+1}=\boldsymbol{r}_k-\boldsymbol{A}\boldsymbol{p}_{k},
- \end{equation*}
-$$
-
Revisiting our first homework
diff --git a/doc/pub/week38/html/week38.html b/doc/pub/week38/html/week38.html
index 332960e54..d78efc01a 100644
--- a/doc/pub/week38/html/week38.html
+++ b/doc/pub/week38/html/week38.html
@@ -273,47 +273,6 @@ div.toc p,a {
'conditions-on-convex-functions'),
('More on convex functions', 2, None, 'more-on-convex-functions'),
('Some simple problems', 2, None, 'some-simple-problems'),
- ('Standard steepest descent',
- 2,
- None,
- 'standard-steepest-descent'),
- ('Gradient method', 2, None, 'gradient-method'),
- ('Steepest descent method', 2, None, 'steepest-descent-method'),
- ('Steepest descent method', 2, None, 'steepest-descent-method'),
- ('Final expressions', 2, None, 'final-expressions'),
- ('Steepest descent example', 2, None, 'steepest-descent-example'),
- ('Conjugate gradient method',
- 2,
- None,
- 'conjugate-gradient-method'),
- ('Conjugate gradient method',
- 2,
- None,
- 'conjugate-gradient-method'),
- ('Conjugate gradient method',
- 2,
- None,
- 'conjugate-gradient-method'),
- ('Conjugate gradient method',
- 2,
- None,
- 'conjugate-gradient-method'),
- ('Conjugate gradient method and iterations',
- 2,
- None,
- 'conjugate-gradient-method-and-iterations'),
- ('Conjugate gradient method',
- 2,
- None,
- 'conjugate-gradient-method'),
- ('Conjugate gradient method',
- 2,
- None,
- 'conjugate-gradient-method'),
- ('Conjugate gradient method',
- 2,
- None,
- 'conjugate-gradient-method'),
('Revisiting our first homework',
2,
None,
@@ -1632,8 +1591,6 @@ X_train, X_test, y_train, y_test = train_tes
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import cross_validate
@@ -1642,7 +1599,6 @@ accuracy = cross_validate(logreg,X_test,y_te
print(accuracy)
print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-
import scikitplot as skplt
y_pred = logreg.predict(X_test)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
@@ -2118,529 +2074,6 @@ is minimal, where \( f \) is convex and differentiable. Then, any point
Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
-
-Standard steepest descent
-
-Before we proceed, we would like to discuss the approach called the
-standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able
-to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).
-
-
-The success of the CG method
-for finding solutions of non-linear problems is based on the theory
-of conjugate gradients for linear systems of equations. It belongs to
-the class of iterative methods for solving problems from linear
-algebra of the type
-
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}.
-\end{equation*}
-$$
-
-In the iterative process we end up with a problem like
-
-$$
-\begin{equation*}
- \boldsymbol{r}= \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x},
-\end{equation*}
-$$
-
-where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.
-
-When we have found the exact solution, \( \boldsymbol{r}=0 \).
-
-
-Gradient method
-
-The residual is zero when we reach the minimum of the quadratic equation
-$$
-\begin{equation*}
- P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b},
-\end{equation*}
-$$
-
-with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and
-symmetric. This defines also the Hessian and we want it to be positive definite.
-
-
-
-Steepest descent method
-
-We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \).
-We can assume without loss of generality that
-
-$$
-\begin{equation*}
-\boldsymbol{x}_0=0,
-\end{equation*}
-$$
-
-or consider the system
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0,
-\end{equation*}
-$$
-
-instead.
-
-
-Steepest descent method
-
-
-
-
One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form
-$$
-\begin{equation*}
- f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n.
-\end{equation*}
-$$
-
-
This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition)
-to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \),
-which equals
-
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b},
-\end{equation*}
-$$
-
-
and
-\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).
-
-
-Final expressions
-
-
-
-
We can compute the residual iteratively as
-$$
-\begin{equation*}
-\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1},
- \end{equation*}
-$$
-
-
which equals
-$$
-\begin{equation*}
-\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k),
- \end{equation*}
-$$
-
-
or
-$$
-\begin{equation*}
-(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k,
- \end{equation*}
-$$
-
-
which gives
-
-$$
-\alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k}
-$$
-
-
leading to the iterative scheme
-$$
-\begin{equation*}
-\boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k},
- \end{equation*}
-$$
-
-Steepest descent example
-
-
-
-
-
-And then as countor plot
-
-
-
-
-Find guesses
-
-
-
-
-Run it!
-
-
-
-
-What happened?
-
-
-
-
-Note that we did only one iteration here. We can easily add more using our previous guesses.
-
-
-Conjugate gradient method
-
-
-
-
In the CG method we define so-called conjugate directions and two vectors
-\( \boldsymbol{s} \) and \( \boldsymbol{t} \)
-are said to be
-conjugate if
-
-$$
-\begin{equation*}
-\boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0.
-\end{equation*}
-$$
-
-
The philosophy of the CG method is to perform searches in various conjugate directions
-of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely
-
-$$
-\begin{equation*}
-\boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0.
-\end{equation*}
-$$
-
-
Two vectors are conjugate if they are orthogonal with respect to
-this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \).
-
-
-Conjugate gradient method
-
-
-
-
An example is given by the eigenvectors of the matrix
-$$
-\begin{equation*}
-\boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j,
-\end{equation*}
-$$
-
-
which is zero unless \( i=j \).
-
-Conjugate gradient method
-
-
-
-
Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size
-\( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector
-
-$$
-\begin{equation*}
-\boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}.
-\end{equation*}
-$$
-
-
We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions.
-Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution
-$ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely
-
-
-$$
-\begin{equation*}
- \boldsymbol{x} = \sum^{n}_{i=1} \alpha_i \boldsymbol{p}_i.
-\end{equation*}
-$$
-
-Conjugate gradient method
-
-
-
-
The coefficients are given by
-$$
-\begin{equation*}
- \mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}.
-\end{equation*}
-$$
-
-
Multiplying with \( \boldsymbol{p}_k^T \) from the left gives
-
-$$
-\begin{equation*}
- \boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{x} = \sum^{n}_{i=1} \alpha_i\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{p}_i= \boldsymbol{p}_k^T \boldsymbol{b},
-\end{equation*}
-$$
-
-
and we can define the coefficients \( \alpha_k \) as
-
-$$
-\begin{equation*}
- \alpha_k = \frac{\boldsymbol{p}_k^T \boldsymbol{b}}{\boldsymbol{p}_k^T \boldsymbol{A} \boldsymbol{p}_k}
-\end{equation*}
-$$
-
-Conjugate gradient method and iterations
-
-
-
-
-
If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully,
-then we may not need all of them to obtain a good approximation to the solution
-\( \boldsymbol{x} \).
-We want to regard the conjugate gradient method as an iterative method.
-This will us to solve systems where \( n \) is so large that the direct
-method would take too much time.
-
-
-
We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \).
-We can assume without loss of generality that
-
-$$
-\begin{equation*}
-\boldsymbol{x}_0=0,
-\end{equation*}
-$$
-
-
or consider the system
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0,
-\end{equation*}
-$$
-
-
instead.
-
-Conjugate gradient method
-
-
-
-
One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form
-$$
-\begin{equation*}
- f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n.
-\end{equation*}
-$$
-
-
This suggests taking the first basis vector \( \boldsymbol{p}_1 \)
-to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \),
-which equals
-
-$$
-\begin{equation*}
-\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b},
-\end{equation*}
-$$
-
-
and
-\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).
-The other vectors in the basis will be conjugate to the gradient,
-hence the name conjugate gradient method.
-
-
-Conjugate gradient method
-
-
-
-
Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step:
-$$
-\begin{equation*}
-\boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k.
-\end{equation*}
-$$
-
-
Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at
-\( \boldsymbol{x}=\boldsymbol{x}_k \),
-so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \).
-Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other,
-so we take the direction closest to the gradient \( \boldsymbol{r}_k \)
-under the conjugacy constraint.
-This gives the following expression
-
-$$
-\begin{equation*}
-\boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k.
-\end{equation*}
-$$
-
-Conjugate gradient method
-
-
-
-
We can also compute the residual iteratively as
-$$
-\begin{equation*}
-\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1},
- \end{equation*}
-$$
-
-
which equals
-$$
-\begin{equation*}
-\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k),
- \end{equation*}
-$$
-
-
or
-$$
-\begin{equation*}
-(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k,
- \end{equation*}
-$$
-
-
which gives
-
-$$
-\begin{equation*}
-\boldsymbol{r}_{k+1}=\boldsymbol{r}_k-\boldsymbol{A}\boldsymbol{p}_{k},
- \end{equation*}
-$$
-
Revisiting our first homework
diff --git a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz
index 58c236f87..4408c6d9b 100644
Binary files a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz and b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz differ
diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb
index 0c004ed0e..b0aca7109 100644
--- a/doc/pub/week38/ipynb/week38.ipynb
+++ b/doc/pub/week38/ipynb/week38.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "3e8a2ce3",
+ "id": "29ef836c",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "10a6370e",
+ "id": "04e43ab6",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
- "id": "056f7021",
+ "id": "874182e4",
"metadata": {
"editable": true
},
@@ -53,7 +53,7 @@
},
{
"cell_type": "markdown",
- "id": "d6830db3",
+ "id": "d0b402a3",
"metadata": {
"editable": true
},
@@ -66,7 +66,7 @@
},
{
"cell_type": "markdown",
- "id": "fc7841e6",
+ "id": "b45b9ff7",
"metadata": {
"editable": true
},
@@ -78,7 +78,7 @@
},
{
"cell_type": "markdown",
- "id": "efbc6f20",
+ "id": "529e022b",
"metadata": {
"editable": true
},
@@ -88,7 +88,7 @@
},
{
"cell_type": "markdown",
- "id": "73a28375",
+ "id": "d0a946a9",
"metadata": {
"editable": true
},
@@ -101,7 +101,7 @@
},
{
"cell_type": "markdown",
- "id": "9e0456b3",
+ "id": "1e389406",
"metadata": {
"editable": true
},
@@ -111,7 +111,7 @@
},
{
"cell_type": "markdown",
- "id": "702bc408",
+ "id": "6beb7daa",
"metadata": {
"editable": true
},
@@ -123,7 +123,7 @@
},
{
"cell_type": "markdown",
- "id": "dac28ddc",
+ "id": "d2adcc96",
"metadata": {
"editable": true
},
@@ -136,7 +136,7 @@
},
{
"cell_type": "markdown",
- "id": "b3c67358",
+ "id": "ea18f58f",
"metadata": {
"editable": true
},
@@ -149,7 +149,7 @@
},
{
"cell_type": "markdown",
- "id": "73f37689",
+ "id": "1d1a3360",
"metadata": {
"editable": true
},
@@ -161,7 +161,7 @@
},
{
"cell_type": "markdown",
- "id": "99cf2ef6",
+ "id": "ea825d7d",
"metadata": {
"editable": true
},
@@ -173,7 +173,7 @@
},
{
"cell_type": "markdown",
- "id": "c5ed9ef9",
+ "id": "5be1e9a8",
"metadata": {
"editable": true
},
@@ -183,7 +183,7 @@
},
{
"cell_type": "markdown",
- "id": "4b3d1402",
+ "id": "e5ab7fb3",
"metadata": {
"editable": true
},
@@ -196,7 +196,7 @@
},
{
"cell_type": "markdown",
- "id": "5289b65c",
+ "id": "51eca8f5",
"metadata": {
"editable": true
},
@@ -208,7 +208,7 @@
},
{
"cell_type": "markdown",
- "id": "706b486a",
+ "id": "bef46cda",
"metadata": {
"editable": true
},
@@ -220,7 +220,7 @@
},
{
"cell_type": "markdown",
- "id": "f62b71be",
+ "id": "cb40ab90",
"metadata": {
"editable": true
},
@@ -245,7 +245,7 @@
},
{
"cell_type": "markdown",
- "id": "2d1c638a",
+ "id": "85735d28",
"metadata": {
"editable": true
},
@@ -261,7 +261,7 @@
},
{
"cell_type": "markdown",
- "id": "11c6d5b3",
+ "id": "8b201ff0",
"metadata": {
"editable": true
},
@@ -277,7 +277,7 @@
},
{
"cell_type": "markdown",
- "id": "971994c8",
+ "id": "ba6fb049",
"metadata": {
"editable": true
},
@@ -291,7 +291,7 @@
},
{
"cell_type": "markdown",
- "id": "88259576",
+ "id": "32493fc3",
"metadata": {
"editable": true
},
@@ -319,7 +319,7 @@
},
{
"cell_type": "markdown",
- "id": "5c4f6ef8",
+ "id": "942d2d71",
"metadata": {
"editable": true
},
@@ -332,7 +332,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "e1d5fcb7",
+ "id": "0822cca6",
"metadata": {
"collapsed": false,
"editable": true
@@ -434,7 +434,7 @@
},
{
"cell_type": "markdown",
- "id": "3333931f",
+ "id": "5e6943d5",
"metadata": {
"editable": true
},
@@ -456,7 +456,7 @@
},
{
"cell_type": "markdown",
- "id": "014b9fa9",
+ "id": "c6f01005",
"metadata": {
"editable": true
},
@@ -482,7 +482,7 @@
},
{
"cell_type": "markdown",
- "id": "1e27dc3e",
+ "id": "65485ae8",
"metadata": {
"editable": true
},
@@ -506,7 +506,7 @@
},
{
"cell_type": "markdown",
- "id": "403f59b9",
+ "id": "77da4608",
"metadata": {
"editable": true
},
@@ -531,7 +531,7 @@
},
{
"cell_type": "markdown",
- "id": "4689115f",
+ "id": "395292f8",
"metadata": {
"editable": true
},
@@ -543,7 +543,7 @@
},
{
"cell_type": "markdown",
- "id": "1c62959a",
+ "id": "783b7c46",
"metadata": {
"editable": true
},
@@ -561,7 +561,7 @@
},
{
"cell_type": "markdown",
- "id": "6afffdf9",
+ "id": "1781e6fd",
"metadata": {
"editable": true
},
@@ -579,7 +579,7 @@
},
{
"cell_type": "markdown",
- "id": "f3e9c250",
+ "id": "e92a522b",
"metadata": {
"editable": true
},
@@ -590,7 +590,7 @@
},
{
"cell_type": "markdown",
- "id": "fd0c7f43",
+ "id": "38f13d47",
"metadata": {
"editable": true
},
@@ -617,7 +617,7 @@
},
{
"cell_type": "markdown",
- "id": "7d842aae",
+ "id": "f0cc1c19",
"metadata": {
"editable": true
},
@@ -630,7 +630,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "ca5e7fcf",
+ "id": "8b9a2f9b",
"metadata": {
"collapsed": false,
"editable": true
@@ -695,7 +695,7 @@
},
{
"cell_type": "markdown",
- "id": "8473515a",
+ "id": "9e40a566",
"metadata": {
"editable": true
},
@@ -708,7 +708,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "24c5579a",
+ "id": "73b773ef",
"metadata": {
"collapsed": false,
"editable": true
@@ -727,7 +727,7 @@
},
{
"cell_type": "markdown",
- "id": "edfc7e09",
+ "id": "5f333675",
"metadata": {
"editable": true
},
@@ -738,7 +738,7 @@
},
{
"cell_type": "markdown",
- "id": "e51b77a1",
+ "id": "1912adab",
"metadata": {
"editable": true
},
@@ -750,7 +750,7 @@
},
{
"cell_type": "markdown",
- "id": "e284bdf6",
+ "id": "2789f0aa",
"metadata": {
"editable": true
},
@@ -769,7 +769,7 @@
},
{
"cell_type": "markdown",
- "id": "bba43329",
+ "id": "077ff376",
"metadata": {
"editable": true
},
@@ -791,7 +791,7 @@
},
{
"cell_type": "markdown",
- "id": "09beca11",
+ "id": "ea6a9677",
"metadata": {
"editable": true
},
@@ -803,7 +803,7 @@
},
{
"cell_type": "markdown",
- "id": "dfac755d",
+ "id": "d5682b86",
"metadata": {
"editable": true
},
@@ -813,7 +813,7 @@
},
{
"cell_type": "markdown",
- "id": "a560feb3",
+ "id": "e7cbcd65",
"metadata": {
"editable": true
},
@@ -826,7 +826,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "b76f36c1",
+ "id": "35f06e8b",
"metadata": {
"collapsed": false,
"editable": true
@@ -891,7 +891,7 @@
},
{
"cell_type": "markdown",
- "id": "c7b9368f",
+ "id": "0d60f84f",
"metadata": {
"editable": true
},
@@ -903,7 +903,7 @@
},
{
"cell_type": "markdown",
- "id": "8b3d6c4d",
+ "id": "b59afebf",
"metadata": {
"editable": true
},
@@ -918,7 +918,7 @@
},
{
"cell_type": "markdown",
- "id": "ec9d13f8",
+ "id": "8c20e9af",
"metadata": {
"editable": true
},
@@ -930,7 +930,7 @@
},
{
"cell_type": "markdown",
- "id": "e64a8477",
+ "id": "6e678787",
"metadata": {
"editable": true
},
@@ -942,7 +942,7 @@
},
{
"cell_type": "markdown",
- "id": "838d72e6",
+ "id": "87315bfa",
"metadata": {
"editable": true
},
@@ -959,7 +959,7 @@
},
{
"cell_type": "markdown",
- "id": "575e2f5c",
+ "id": "b35af430",
"metadata": {
"editable": true
},
@@ -973,7 +973,7 @@
},
{
"cell_type": "markdown",
- "id": "f604f919",
+ "id": "95642c01",
"metadata": {
"editable": true
},
@@ -983,7 +983,7 @@
},
{
"cell_type": "markdown",
- "id": "0758a737",
+ "id": "216ec0e8",
"metadata": {
"editable": true
},
@@ -995,7 +995,7 @@
},
{
"cell_type": "markdown",
- "id": "17f088cc",
+ "id": "17a272ea",
"metadata": {
"editable": true
},
@@ -1007,7 +1007,7 @@
},
{
"cell_type": "markdown",
- "id": "bab96f9f",
+ "id": "4a947d2e",
"metadata": {
"editable": true
},
@@ -1019,7 +1019,7 @@
},
{
"cell_type": "markdown",
- "id": "a90dfd27",
+ "id": "8af36a82",
"metadata": {
"editable": true
},
@@ -1030,7 +1030,7 @@
},
{
"cell_type": "markdown",
- "id": "95190f09",
+ "id": "9ee722ed",
"metadata": {
"editable": true
},
@@ -1042,7 +1042,7 @@
},
{
"cell_type": "markdown",
- "id": "6f666380",
+ "id": "ec4a3ac0",
"metadata": {
"editable": true
},
@@ -1053,7 +1053,7 @@
},
{
"cell_type": "markdown",
- "id": "5a6168f7",
+ "id": "6bb4ca7d",
"metadata": {
"editable": true
},
@@ -1069,7 +1069,7 @@
},
{
"cell_type": "markdown",
- "id": "77ed6009",
+ "id": "1a91b11b",
"metadata": {
"editable": true
},
@@ -1081,7 +1081,7 @@
},
{
"cell_type": "markdown",
- "id": "858db43f",
+ "id": "6d4ad1b7",
"metadata": {
"editable": true
},
@@ -1091,7 +1091,7 @@
},
{
"cell_type": "markdown",
- "id": "db45b51a",
+ "id": "dccd3d6f",
"metadata": {
"editable": true
},
@@ -1103,7 +1103,7 @@
},
{
"cell_type": "markdown",
- "id": "131521d4",
+ "id": "be548919",
"metadata": {
"editable": true
},
@@ -1118,7 +1118,7 @@
},
{
"cell_type": "markdown",
- "id": "8f32ba93",
+ "id": "59939b49",
"metadata": {
"editable": true
},
@@ -1130,7 +1130,7 @@
},
{
"cell_type": "markdown",
- "id": "4acacaf4",
+ "id": "06d249a9",
"metadata": {
"editable": true
},
@@ -1141,7 +1141,7 @@
},
{
"cell_type": "markdown",
- "id": "4abab487",
+ "id": "d98943cd",
"metadata": {
"editable": true
},
@@ -1153,7 +1153,7 @@
},
{
"cell_type": "markdown",
- "id": "9fe4e3fc",
+ "id": "ca77eee0",
"metadata": {
"editable": true
},
@@ -1165,7 +1165,7 @@
},
{
"cell_type": "markdown",
- "id": "cb68654c",
+ "id": "555522ce",
"metadata": {
"editable": true
},
@@ -1177,7 +1177,7 @@
},
{
"cell_type": "markdown",
- "id": "c5978294",
+ "id": "fa6aa07e",
"metadata": {
"editable": true
},
@@ -1187,7 +1187,7 @@
},
{
"cell_type": "markdown",
- "id": "fd62a946",
+ "id": "e7a47c69",
"metadata": {
"editable": true
},
@@ -1199,7 +1199,7 @@
},
{
"cell_type": "markdown",
- "id": "0069180b",
+ "id": "1152b511",
"metadata": {
"editable": true
},
@@ -1213,7 +1213,7 @@
},
{
"cell_type": "markdown",
- "id": "22e5090a",
+ "id": "e21201d4",
"metadata": {
"editable": true
},
@@ -1225,7 +1225,7 @@
},
{
"cell_type": "markdown",
- "id": "3684cad5",
+ "id": "bf0b1ce2",
"metadata": {
"editable": true
},
@@ -1235,7 +1235,7 @@
},
{
"cell_type": "markdown",
- "id": "e8cf0493",
+ "id": "bc15aa1e",
"metadata": {
"editable": true
},
@@ -1247,7 +1247,7 @@
},
{
"cell_type": "markdown",
- "id": "c5eacaff",
+ "id": "fe389fa1",
"metadata": {
"editable": true
},
@@ -1257,7 +1257,7 @@
},
{
"cell_type": "markdown",
- "id": "aabcb0d9",
+ "id": "e4e1cbfd",
"metadata": {
"editable": true
},
@@ -1269,7 +1269,7 @@
},
{
"cell_type": "markdown",
- "id": "c2a20da4",
+ "id": "23152f67",
"metadata": {
"editable": true
},
@@ -1280,7 +1280,7 @@
},
{
"cell_type": "markdown",
- "id": "edd2eb03",
+ "id": "0b904be3",
"metadata": {
"editable": true
},
@@ -1303,7 +1303,7 @@
},
{
"cell_type": "markdown",
- "id": "5fa9456e",
+ "id": "2d759e0f",
"metadata": {
"editable": true
},
@@ -1315,7 +1315,7 @@
},
{
"cell_type": "markdown",
- "id": "94a7740b",
+ "id": "fab6c23c",
"metadata": {
"editable": true
},
@@ -1325,7 +1325,7 @@
},
{
"cell_type": "markdown",
- "id": "f7ab0f4e",
+ "id": "aa3f9525",
"metadata": {
"editable": true
},
@@ -1337,7 +1337,7 @@
},
{
"cell_type": "markdown",
- "id": "dea5150b",
+ "id": "cae6b286",
"metadata": {
"editable": true
},
@@ -1354,7 +1354,7 @@
},
{
"cell_type": "markdown",
- "id": "f0083e77",
+ "id": "4281725f",
"metadata": {
"editable": true
},
@@ -1364,7 +1364,7 @@
},
{
"cell_type": "markdown",
- "id": "d0f0ed57",
+ "id": "33e897bb",
"metadata": {
"editable": true
},
@@ -1383,7 +1383,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "200f832d",
+ "id": "9afb8352",
"metadata": {
"collapsed": false,
"editable": true
@@ -1438,7 +1438,7 @@
},
{
"cell_type": "markdown",
- "id": "727bec9c",
+ "id": "e98a4468",
"metadata": {
"editable": true
},
@@ -1450,7 +1450,7 @@
},
{
"cell_type": "markdown",
- "id": "863d7406",
+ "id": "c9f83bc2",
"metadata": {
"editable": true
},
@@ -1465,7 +1465,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "b3376d03",
+ "id": "db311e69",
"metadata": {
"collapsed": false,
"editable": true
@@ -1518,7 +1518,7 @@
},
{
"cell_type": "markdown",
- "id": "8dbb6c35",
+ "id": "ceec4fb1",
"metadata": {
"editable": true
},
@@ -1533,7 +1533,7 @@
},
{
"cell_type": "markdown",
- "id": "ff1937e5",
+ "id": "ad0c9dbb",
"metadata": {
"editable": true
},
@@ -1553,7 +1553,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "4ac3c78f",
+ "id": "11b6cfab",
"metadata": {
"collapsed": false,
"editable": true
@@ -1607,7 +1607,7 @@
},
{
"cell_type": "markdown",
- "id": "46d97ba4",
+ "id": "69430481",
"metadata": {
"editable": true
},
@@ -1622,7 +1622,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "be6b000e",
+ "id": "74ca9036",
"metadata": {
"collapsed": false,
"editable": true
@@ -1649,7 +1649,7 @@
},
{
"cell_type": "markdown",
- "id": "57d57c49",
+ "id": "f8487661",
"metadata": {
"editable": true
},
@@ -1663,7 +1663,7 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "0c95c073",
+ "id": "e6d68d3e",
"metadata": {
"collapsed": false,
"editable": true
@@ -1708,7 +1708,7 @@
},
{
"cell_type": "markdown",
- "id": "2cf9ba9b",
+ "id": "49a15360",
"metadata": {
"editable": true
},
@@ -1733,7 +1733,7 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "0c3677b2",
+ "id": "3870dc95",
"metadata": {
"collapsed": false,
"editable": true
@@ -1745,7 +1745,7 @@
},
{
"cell_type": "markdown",
- "id": "2a557cc6",
+ "id": "55b78595",
"metadata": {
"editable": true
},
@@ -1756,7 +1756,7 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "7d9f248a",
+ "id": "e0913cac",
"metadata": {
"collapsed": false,
"editable": true
@@ -1768,7 +1768,7 @@
},
{
"cell_type": "markdown",
- "id": "1c5a3d0b",
+ "id": "7b21be8d",
"metadata": {
"editable": true
},
@@ -1781,7 +1781,7 @@
},
{
"cell_type": "markdown",
- "id": "b17ae9d1",
+ "id": "b34571be",
"metadata": {
"editable": true
},
@@ -1792,7 +1792,7 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "09e1b104",
+ "id": "93d889a8",
"metadata": {
"collapsed": false,
"editable": true
@@ -1814,8 +1814,6 @@
"# Logistic Regression\n",
"logreg = LogisticRegression(solver='lbfgs')\n",
"logreg.fit(X_train, y_train)\n",
- "print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
- "\n",
"\n",
"from sklearn.preprocessing import LabelEncoder\n",
"from sklearn.model_selection import cross_validate\n",
@@ -1824,7 +1822,6 @@
"print(accuracy)\n",
"print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
"\n",
- "\n",
"import scikitplot as skplt\n",
"y_pred = logreg.predict(X_test)\n",
"skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
@@ -1838,7 +1835,7 @@
},
{
"cell_type": "markdown",
- "id": "26073b72",
+ "id": "438e33e3",
"metadata": {
"editable": true
},
@@ -1859,7 +1856,7 @@
},
{
"cell_type": "markdown",
- "id": "7e8c8e29",
+ "id": "252cd985",
"metadata": {
"editable": true
},
@@ -1876,7 +1873,7 @@
},
{
"cell_type": "markdown",
- "id": "61bbdf38",
+ "id": "c8eade61",
"metadata": {
"editable": true
},
@@ -1891,7 +1888,7 @@
},
{
"cell_type": "markdown",
- "id": "3bc34603",
+ "id": "dea75e60",
"metadata": {
"editable": true
},
@@ -1901,7 +1898,7 @@
},
{
"cell_type": "markdown",
- "id": "790c6250",
+ "id": "aed7e5af",
"metadata": {
"editable": true
},
@@ -1917,7 +1914,7 @@
},
{
"cell_type": "markdown",
- "id": "749048ff",
+ "id": "0f185516",
"metadata": {
"editable": true
},
@@ -1929,7 +1926,7 @@
},
{
"cell_type": "markdown",
- "id": "6851aec4",
+ "id": "720d1af1",
"metadata": {
"editable": true
},
@@ -1940,7 +1937,7 @@
},
{
"cell_type": "markdown",
- "id": "67215af8",
+ "id": "e3203d75",
"metadata": {
"editable": true
},
@@ -1952,7 +1949,7 @@
},
{
"cell_type": "markdown",
- "id": "5ebec84d",
+ "id": "a33dd1f0",
"metadata": {
"editable": true
},
@@ -1962,7 +1959,7 @@
},
{
"cell_type": "markdown",
- "id": "ff0ef66d",
+ "id": "e39fa431",
"metadata": {
"editable": true
},
@@ -1976,7 +1973,7 @@
},
{
"cell_type": "markdown",
- "id": "7d831a76",
+ "id": "b38d6957",
"metadata": {
"editable": true
},
@@ -1988,7 +1985,7 @@
},
{
"cell_type": "markdown",
- "id": "3cf22961",
+ "id": "19a4fce0",
"metadata": {
"editable": true
},
@@ -1998,7 +1995,7 @@
},
{
"cell_type": "markdown",
- "id": "ed025768",
+ "id": "45d09cf0",
"metadata": {
"editable": true
},
@@ -2010,7 +2007,7 @@
},
{
"cell_type": "markdown",
- "id": "8d38644e",
+ "id": "e986bb71",
"metadata": {
"editable": true
},
@@ -2022,7 +2019,7 @@
},
{
"cell_type": "markdown",
- "id": "f642337a",
+ "id": "af0c244c",
"metadata": {
"editable": true
},
@@ -2042,7 +2039,7 @@
},
{
"cell_type": "markdown",
- "id": "30279543",
+ "id": "9a061289",
"metadata": {
"editable": true
},
@@ -2058,7 +2055,7 @@
},
{
"cell_type": "markdown",
- "id": "ac298d9d",
+ "id": "afb03ff5",
"metadata": {
"editable": true
},
@@ -2074,7 +2071,7 @@
},
{
"cell_type": "markdown",
- "id": "4a0fe169",
+ "id": "304b5ea7",
"metadata": {
"editable": true
},
@@ -2085,7 +2082,7 @@
},
{
"cell_type": "markdown",
- "id": "90b8af67",
+ "id": "578c8540",
"metadata": {
"editable": true
},
@@ -2097,7 +2094,7 @@
},
{
"cell_type": "markdown",
- "id": "1a5da044",
+ "id": "e8750d8f",
"metadata": {
"editable": true
},
@@ -2107,7 +2104,7 @@
},
{
"cell_type": "markdown",
- "id": "ff0ae3a0",
+ "id": "c306bf6a",
"metadata": {
"editable": true
},
@@ -2119,7 +2116,7 @@
},
{
"cell_type": "markdown",
- "id": "6069e3a1",
+ "id": "b247f287",
"metadata": {
"editable": true
},
@@ -2129,7 +2126,7 @@
},
{
"cell_type": "markdown",
- "id": "582b5ce3",
+ "id": "40851e5b",
"metadata": {
"editable": true
},
@@ -2141,7 +2138,7 @@
},
{
"cell_type": "markdown",
- "id": "d6c0830c",
+ "id": "d01734da",
"metadata": {
"editable": true
},
@@ -2163,7 +2160,7 @@
},
{
"cell_type": "markdown",
- "id": "b50cb722",
+ "id": "1475257c",
"metadata": {
"editable": true
},
@@ -2176,7 +2173,7 @@
},
{
"cell_type": "markdown",
- "id": "8af1d1b3",
+ "id": "bf260ca5",
"metadata": {
"editable": true
},
@@ -2189,7 +2186,7 @@
},
{
"cell_type": "markdown",
- "id": "0fb6a685",
+ "id": "1a6a0163",
"metadata": {
"editable": true
},
@@ -2199,7 +2196,7 @@
},
{
"cell_type": "markdown",
- "id": "cbcbbcb6",
+ "id": "75428d23",
"metadata": {
"editable": true
},
@@ -2217,7 +2214,7 @@
},
{
"cell_type": "markdown",
- "id": "44b2eac6",
+ "id": "3422e608",
"metadata": {
"editable": true
},
@@ -2227,7 +2224,7 @@
},
{
"cell_type": "markdown",
- "id": "61a24a60",
+ "id": "cea80cf2",
"metadata": {
"editable": true
},
@@ -2242,7 +2239,7 @@
},
{
"cell_type": "markdown",
- "id": "885c0e0c",
+ "id": "1d9bd9a3",
"metadata": {
"editable": true
},
@@ -2252,7 +2249,7 @@
},
{
"cell_type": "markdown",
- "id": "438ac583",
+ "id": "60b2e14c",
"metadata": {
"editable": true
},
@@ -2266,7 +2263,7 @@
},
{
"cell_type": "markdown",
- "id": "3df2e7b9",
+ "id": "2a422ee3",
"metadata": {
"editable": true
},
@@ -2276,7 +2273,7 @@
},
{
"cell_type": "markdown",
- "id": "86053308",
+ "id": "c44dd1ba",
"metadata": {
"editable": true
},
@@ -2290,7 +2287,7 @@
},
{
"cell_type": "markdown",
- "id": "6bee4f6b",
+ "id": "67b8a115",
"metadata": {
"editable": true
},
@@ -2305,7 +2302,7 @@
},
{
"cell_type": "markdown",
- "id": "2fff8043",
+ "id": "09510bcc",
"metadata": {
"editable": true
},
@@ -2322,7 +2319,7 @@
},
{
"cell_type": "markdown",
- "id": "4eb6e3e7",
+ "id": "50c80755",
"metadata": {
"editable": true
},
@@ -2334,7 +2331,7 @@
},
{
"cell_type": "markdown",
- "id": "de881951",
+ "id": "1fbf7076",
"metadata": {
"editable": true
},
@@ -2348,7 +2345,7 @@
},
{
"cell_type": "markdown",
- "id": "8a41b62a",
+ "id": "2d22e42e",
"metadata": {
"editable": true
},
@@ -2363,7 +2360,7 @@
},
{
"cell_type": "markdown",
- "id": "f4f00af6",
+ "id": "107c0c93",
"metadata": {
"editable": true
},
@@ -2375,7 +2372,7 @@
},
{
"cell_type": "markdown",
- "id": "537efe83",
+ "id": "a94c6b7d",
"metadata": {
"editable": true
},
@@ -2386,7 +2383,7 @@
},
{
"cell_type": "markdown",
- "id": "028f751a",
+ "id": "415e7088",
"metadata": {
"editable": true
},
@@ -2414,7 +2411,7 @@
},
{
"cell_type": "markdown",
- "id": "f0a5dfe5",
+ "id": "1fcd9773",
"metadata": {
"editable": true
},
@@ -2436,7 +2433,7 @@
},
{
"cell_type": "markdown",
- "id": "6cf0f7ff",
+ "id": "7bc29e67",
"metadata": {
"editable": true
},
@@ -2453,7 +2450,7 @@
},
{
"cell_type": "markdown",
- "id": "54aa7188",
+ "id": "2591623a",
"metadata": {
"editable": true
},
@@ -2468,7 +2465,7 @@
},
{
"cell_type": "markdown",
- "id": "fa9ecd28",
+ "id": "74941000",
"metadata": {
"editable": true
},
@@ -2478,7 +2475,7 @@
},
{
"cell_type": "markdown",
- "id": "ddb180ec",
+ "id": "e8a83c98",
"metadata": {
"editable": true
},
@@ -2494,7 +2491,7 @@
},
{
"cell_type": "markdown",
- "id": "03802ec9",
+ "id": "0a28a0fc",
"metadata": {
"editable": true
},
@@ -2506,7 +2503,7 @@
},
{
"cell_type": "markdown",
- "id": "5f69d611",
+ "id": "1186db52",
"metadata": {
"editable": true
},
@@ -2517,7 +2514,7 @@
},
{
"cell_type": "markdown",
- "id": "f66d3fe2",
+ "id": "86f167a3",
"metadata": {
"editable": true
},
@@ -2529,7 +2526,7 @@
},
{
"cell_type": "markdown",
- "id": "ce5bdb91",
+ "id": "559c083b",
"metadata": {
"editable": true
},
@@ -2539,7 +2536,7 @@
},
{
"cell_type": "markdown",
- "id": "8baf2519",
+ "id": "527ecec2",
"metadata": {
"editable": true
},
@@ -2553,7 +2550,7 @@
},
{
"cell_type": "markdown",
- "id": "aa7979dc",
+ "id": "2b43706a",
"metadata": {
"editable": true
},
@@ -2565,7 +2562,7 @@
},
{
"cell_type": "markdown",
- "id": "3dc74370",
+ "id": "3f9cc07b",
"metadata": {
"editable": true
},
@@ -2575,7 +2572,7 @@
},
{
"cell_type": "markdown",
- "id": "6837858a",
+ "id": "efb2648e",
"metadata": {
"editable": true
},
@@ -2587,7 +2584,7 @@
},
{
"cell_type": "markdown",
- "id": "200dbd05",
+ "id": "e95e3abc",
"metadata": {
"editable": true
},
@@ -2599,7 +2596,7 @@
},
{
"cell_type": "markdown",
- "id": "81a0efac",
+ "id": "57b7e63c",
"metadata": {
"editable": true
},
@@ -2621,7 +2618,7 @@
},
{
"cell_type": "markdown",
- "id": "dc98f867",
+ "id": "b9f8d543",
"metadata": {
"editable": true
},
@@ -2633,7 +2630,7 @@
},
{
"cell_type": "markdown",
- "id": "f684079d",
+ "id": "a6e51c0c",
"metadata": {
"editable": true
},
@@ -2670,7 +2667,7 @@
},
{
"cell_type": "markdown",
- "id": "0d62d282",
+ "id": "3a1c6ae4",
"metadata": {
"editable": true
},
@@ -2698,7 +2695,7 @@
},
{
"cell_type": "markdown",
- "id": "2635310e",
+ "id": "e6acadbb",
"metadata": {
"editable": true
},
@@ -2728,962 +2725,7 @@
},
{
"cell_type": "markdown",
- "id": "2a5fca79",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Standard steepest descent\n",
- "\n",
- "Before we proceed, we would like to discuss the approach called the\n",
- "**standard Steepest descent** (different from the above steepest descent discussion), which again leads to us having to be able\n",
- "to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).\n",
- "\n",
- "[The success of the CG method](https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf)\n",
- "for finding solutions of non-linear problems is based on the theory\n",
- "of conjugate gradients for linear systems of equations. It belongs to\n",
- "the class of iterative methods for solving problems from linear\n",
- "algebra of the type"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c46599ca",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2d6ae580",
- "metadata": {
- "editable": true
- },
- "source": [
- "In the iterative process we end up with a problem like"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cc7847ba",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eaaf8092",
- "metadata": {
- "editable": true
- },
- "source": [
- "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n",
- "\n",
- "When we have found the exact solution, $\\boldsymbol{r}=0$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "884cce89",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Gradient method\n",
- "\n",
- "The residual is zero when we reach the minimum of the quadratic equation"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "86f1f8a0",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f9bbae9c",
- "metadata": {
- "editable": true
- },
- "source": [
- "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n",
- "symmetric. This defines also the Hessian and we want it to be positive definite."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "19a06c73",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Steepest descent method\n",
- "\n",
- "We denote the initial guess for $\\boldsymbol{x}$ as $\\boldsymbol{x}_0$. \n",
- "We can assume without loss of generality that"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f2786bbd",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{x}_0=0,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "14604a52",
- "metadata": {
- "editable": true
- },
- "source": [
- "or consider the system"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "21746c84",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5fc6a840",
- "metadata": {
- "editable": true
- },
- "source": [
- "instead."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "dcf4611f",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Steepest descent method\n",
- "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f5fbcc70",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bd45a6f8",
- "metadata": {
- "editable": true
- },
- "source": [
- "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n",
- "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n",
- "which equals"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a69742db",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c0a824ae",
- "metadata": {
- "editable": true
- },
- "source": [
- "and \n",
- "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1c2ed997",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Final expressions\n",
- "We can compute the residual iteratively as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "930d2097",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f8ac4d3d",
- "metadata": {
- "editable": true
- },
- "source": [
- "which equals"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2a34d4d6",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "db0f4ce3",
- "metadata": {
- "editable": true
- },
- "source": [
- "or"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ddb63da0",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c84ab2fe",
- "metadata": {
- "editable": true
- },
- "source": [
- "which gives"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b4aad6a5",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e462d35e",
- "metadata": {
- "editable": true
- },
- "source": [
- "leading to the iterative scheme"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "33ed7568",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k-\\alpha_k\\boldsymbol{r}_{k},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5e58ef52",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Steepest descent example"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "99362113",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "import numpy.linalg as la\n",
- "\n",
- "import scipy.optimize as sopt\n",
- "\n",
- "import matplotlib.pyplot as pt\n",
- "from mpl_toolkits.mplot3d import axes3d\n",
- "\n",
- "def f(x):\n",
- " return x[0]**2 + 3.0*x[1]**2\n",
- "\n",
- "def df(x):\n",
- " return np.array([2*x[0], 6*x[1]])\n",
- "\n",
- "fig = pt.figure()\n",
- "ax = fig.gca(projection=\"3d\")\n",
- "\n",
- "xmesh, ymesh = np.mgrid[-3:3:50j,-3:3:50j]\n",
- "fmesh = f(np.array([xmesh, ymesh]))\n",
- "ax.plot_surface(xmesh, ymesh, fmesh)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b1d595f9",
- "metadata": {
- "editable": true
- },
- "source": [
- "And then as countor plot"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "id": "b94a2b76",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "pt.axis(\"equal\")\n",
- "pt.contour(xmesh, ymesh, fmesh)\n",
- "guesses = [np.array([2, 2./5])]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9281e4f3",
- "metadata": {
- "editable": true
- },
- "source": [
- "Find guesses"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "e7d524ac",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "x = guesses[-1]\n",
- "s = -df(x)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "472cd65c",
- "metadata": {
- "editable": true
- },
- "source": [
- "Run it!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "0f5375ea",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "def f1d(alpha):\n",
- " return f(x + alpha*s)\n",
- "\n",
- "alpha_opt = sopt.golden(f1d)\n",
- "next_guess = x + alpha_opt * s\n",
- "guesses.append(next_guess)\n",
- "print(next_guess)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ee76b98e",
- "metadata": {
- "editable": true
- },
- "source": [
- "What happened?"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "59ee63a9",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
- "source": [
- "pt.axis(\"equal\")\n",
- "pt.contour(xmesh, ymesh, fmesh, 50)\n",
- "it_array = np.array(guesses)\n",
- "pt.plot(it_array.T[0], it_array.T[1], \"x-\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cacf3aff",
- "metadata": {
- "editable": true
- },
- "source": [
- "Note that we did only one iteration here. We can easily add more using our previous guesses."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ee1f0dda",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method\n",
- "In the CG method we define so-called conjugate directions and two vectors \n",
- "$\\boldsymbol{s}$ and $\\boldsymbol{t}$\n",
- "are said to be\n",
- "conjugate if"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c066ec75",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "84693748",
- "metadata": {
- "editable": true
- },
- "source": [
- "The philosophy of the CG method is to perform searches in various conjugate directions\n",
- "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d68f83df",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "de29a396",
- "metadata": {
- "editable": true
- },
- "source": [
- "Two vectors are conjugate if they are orthogonal with respect to \n",
- "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8cbe6e34",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method\n",
- "An example is given by the eigenvectors of the matrix"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "95e275a9",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f904c826",
- "metadata": {
- "editable": true
- },
- "source": [
- "which is zero unless $i=j$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "591a79b9",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method\n",
- "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n",
- "$n\\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "200883b7",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "06e90b28",
- "metadata": {
- "editable": true
- },
- "source": [
- "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n",
- "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n",
- "$ \\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}$ in this basis, namely"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cd80d48f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "99b996bd",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method\n",
- "The coefficients are given by"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1a4c81c2",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4e883b4d",
- "metadata": {
- "editable": true
- },
- "source": [
- "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e6ea38e1",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a32832f4",
- "metadata": {
- "editable": true
- },
- "source": [
- "and we can define the coefficients $\\alpha_k$ as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f209b17d",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cdc2bb47",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method and iterations\n",
- "\n",
- "If we choose the conjugate vectors $\\boldsymbol{p}_k$ carefully, \n",
- "then we may not need all of them to obtain a good approximation to the solution \n",
- "$\\boldsymbol{x}$. \n",
- "We want to regard the conjugate gradient method as an iterative method. \n",
- "This will us to solve systems where $n$ is so large that the direct \n",
- "method would take too much time.\n",
- "\n",
- "We denote the initial guess for $\\boldsymbol{x}$ as $\\boldsymbol{x}_0$. \n",
- "We can assume without loss of generality that"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e2fa3248",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{x}_0=0,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b71a371b",
- "metadata": {
- "editable": true
- },
- "source": [
- "or consider the system"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "86c2fb96",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "20c5d2e3",
- "metadata": {
- "editable": true
- },
- "source": [
- "instead."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ab14a14a",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method\n",
- "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "613df879",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6c6ae3a7",
- "metadata": {
- "editable": true
- },
- "source": [
- "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n",
- "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n",
- "which equals"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3b34cd25",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "af8e9fd5",
- "metadata": {
- "editable": true
- },
- "source": [
- "and \n",
- "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n",
- "The other vectors in the basis will be conjugate to the gradient, \n",
- "hence the name conjugate gradient method."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "54969b51",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method\n",
- "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "967b98dd",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bd4ce3fa",
- "metadata": {
- "editable": true
- },
- "source": [
- "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n",
- "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n",
- "so the gradient descent method would be to move in the direction $\\boldsymbol{r}_k$. \n",
- "Here, we insist that the directions $\\boldsymbol{p}_k$ are conjugate to each other, \n",
- "so we take the direction closest to the gradient $\\boldsymbol{r}_k$ \n",
- "under the conjugacy constraint. \n",
- "This gives the following expression"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "82c78ed0",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d887c09f",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Conjugate gradient method\n",
- "We can also compute the residual iteratively as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a0773935",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "da7a95c7",
- "metadata": {
- "editable": true
- },
- "source": [
- "which equals"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f132568f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fbce94ad",
- "metadata": {
- "editable": true
- },
- "source": [
- "or"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fee1c104",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "223f6b07",
- "metadata": {
- "editable": true
- },
- "source": [
- "which gives"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "69eec27b",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7340c9bf",
+ "id": "0fa71cae",
"metadata": {
"editable": true
},
@@ -3706,8 +2748,8 @@
},
{
"cell_type": "code",
- "execution_count": 18,
- "id": "f4d4bf5a",
+ "execution_count": 13,
+ "id": "3657e008",
"metadata": {
"collapsed": false,
"editable": true
@@ -3720,7 +2762,7 @@
},
{
"cell_type": "markdown",
- "id": "410091d3",
+ "id": "5cf5087d",
"metadata": {
"editable": true
},
@@ -3731,7 +2773,7 @@
},
{
"cell_type": "markdown",
- "id": "513abc7d",
+ "id": "665338d9",
"metadata": {
"editable": true
},
@@ -3743,7 +2785,7 @@
},
{
"cell_type": "markdown",
- "id": "04836def",
+ "id": "12d24ce4",
"metadata": {
"editable": true
},
@@ -3753,7 +2795,7 @@
},
{
"cell_type": "markdown",
- "id": "ea14f8a5",
+ "id": "e1ae1307",
"metadata": {
"editable": true
},
@@ -3765,7 +2807,7 @@
},
{
"cell_type": "markdown",
- "id": "10781493",
+ "id": "6342baf4",
"metadata": {
"editable": true
},
@@ -3779,7 +2821,7 @@
},
{
"cell_type": "markdown",
- "id": "cb714abf",
+ "id": "7c27ddb3",
"metadata": {
"editable": true
},
@@ -3795,7 +2837,7 @@
},
{
"cell_type": "markdown",
- "id": "ef8678d0",
+ "id": "4f73b206",
"metadata": {
"editable": true
},
@@ -3805,7 +2847,7 @@
},
{
"cell_type": "markdown",
- "id": "1cab181a",
+ "id": "16fbc3dd",
"metadata": {
"editable": true
},
@@ -3817,7 +2859,7 @@
},
{
"cell_type": "markdown",
- "id": "deb43b3c",
+ "id": "82dca579",
"metadata": {
"editable": true
},
@@ -3827,7 +2869,7 @@
},
{
"cell_type": "markdown",
- "id": "0ce616cb",
+ "id": "87f38e83",
"metadata": {
"editable": true
},
@@ -3839,7 +2881,7 @@
},
{
"cell_type": "markdown",
- "id": "9c8e423d",
+ "id": "d8f707ab",
"metadata": {
"editable": true
},
@@ -3853,7 +2895,7 @@
},
{
"cell_type": "markdown",
- "id": "713ee38d",
+ "id": "51de9ac3",
"metadata": {
"editable": true
},
@@ -3863,7 +2905,7 @@
},
{
"cell_type": "markdown",
- "id": "2c1886d0",
+ "id": "0bd452e8",
"metadata": {
"editable": true
},
@@ -3874,7 +2916,7 @@
},
{
"cell_type": "markdown",
- "id": "7c2d4d10",
+ "id": "43da0408",
"metadata": {
"editable": true
},
@@ -3889,7 +2931,7 @@
},
{
"cell_type": "markdown",
- "id": "7926ec5e",
+ "id": "98adc1ed",
"metadata": {
"editable": true
},
@@ -3899,7 +2941,7 @@
},
{
"cell_type": "markdown",
- "id": "58ec1ade",
+ "id": "c13bccab",
"metadata": {
"editable": true
},
@@ -3911,7 +2953,7 @@
},
{
"cell_type": "markdown",
- "id": "c15d194c",
+ "id": "459b1374",
"metadata": {
"editable": true
},
@@ -3923,7 +2965,7 @@
},
{
"cell_type": "markdown",
- "id": "6c11b0c2",
+ "id": "75dd7666",
"metadata": {
"editable": true
},
@@ -3938,7 +2980,7 @@
},
{
"cell_type": "markdown",
- "id": "3cb7b310",
+ "id": "778e9848",
"metadata": {
"editable": true
},
@@ -3950,8 +2992,8 @@
},
{
"cell_type": "code",
- "execution_count": 19,
- "id": "14bda9e8",
+ "execution_count": 14,
+ "id": "52f85a1c",
"metadata": {
"collapsed": false,
"editable": true
@@ -4008,7 +3050,7 @@
},
{
"cell_type": "markdown",
- "id": "3a190974",
+ "id": "fc43d3cf",
"metadata": {
"editable": true
},
@@ -4018,8 +3060,8 @@
},
{
"cell_type": "code",
- "execution_count": 20,
- "id": "b416370f",
+ "execution_count": 15,
+ "id": "2e5876fe",
"metadata": {
"collapsed": false,
"editable": true
@@ -4046,7 +3088,7 @@
},
{
"cell_type": "markdown",
- "id": "83ad723b",
+ "id": "3a778be9",
"metadata": {
"editable": true
},
@@ -4058,7 +3100,7 @@
},
{
"cell_type": "markdown",
- "id": "8d0ffdd9",
+ "id": "c8ea3390",
"metadata": {
"editable": true
},
@@ -4070,7 +3112,7 @@
},
{
"cell_type": "markdown",
- "id": "7b2d6354",
+ "id": "f469189c",
"metadata": {
"editable": true
},
@@ -4080,7 +3122,7 @@
},
{
"cell_type": "markdown",
- "id": "3058d881",
+ "id": "0d2af466",
"metadata": {
"editable": true
},
@@ -4094,7 +3136,7 @@
},
{
"cell_type": "markdown",
- "id": "02cbd1d6",
+ "id": "19c91458",
"metadata": {
"editable": true
},
@@ -4104,7 +3146,7 @@
},
{
"cell_type": "markdown",
- "id": "7e979dfa",
+ "id": "36048e3f",
"metadata": {
"editable": true
},
@@ -4116,7 +3158,7 @@
},
{
"cell_type": "markdown",
- "id": "4fd27d01",
+ "id": "83d71583",
"metadata": {
"editable": true
},
@@ -4127,7 +3169,7 @@
},
{
"cell_type": "markdown",
- "id": "f868972a",
+ "id": "2c5fe2ce",
"metadata": {
"editable": true
},
@@ -4142,7 +3184,7 @@
},
{
"cell_type": "markdown",
- "id": "5a3306bf",
+ "id": "f7972c71",
"metadata": {
"editable": true
},
@@ -4156,7 +3198,7 @@
},
{
"cell_type": "markdown",
- "id": "83787d22",
+ "id": "5c018560",
"metadata": {
"editable": true
},
@@ -4166,8 +3208,8 @@
},
{
"cell_type": "code",
- "execution_count": 21,
- "id": "145da03c",
+ "execution_count": 16,
+ "id": "e1570e58",
"metadata": {
"collapsed": false,
"editable": true
@@ -4228,7 +3270,7 @@
},
{
"cell_type": "markdown",
- "id": "a1a7bf77",
+ "id": "e1a6b8fd",
"metadata": {
"editable": true
},
@@ -4250,7 +3292,7 @@
},
{
"cell_type": "markdown",
- "id": "6f6dc52d",
+ "id": "38ac0b06",
"metadata": {
"editable": true
},
diff --git a/doc/src/week38/week38.do.txt b/doc/src/week38/week38.do.txt
index fbd74b68c..65e4ae189 100644
--- a/doc/src/week38/week38.do.txt
+++ b/doc/src/week38/week38.do.txt
@@ -1050,8 +1050,6 @@ print(X_test.shape)
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
logreg.fit(X_train, y_train)
-print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import cross_validate
@@ -1060,7 +1058,6 @@ accuracy = cross_validate(logreg,X_test,y_test,cv=10)['test_score']
print(accuracy)
print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
-
import scikitplot as skplt
y_pred = logreg.predict(X_test)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
@@ -1527,374 +1524,6 @@ Using the definition of convexity, try to show that a function satisfying the pr
-!split
-===== Standard steepest descent =====
-
-
-Before we proceed, we would like to discuss the approach called the
-_standard Steepest descent_ (different from the above steepest descent discussion), which again leads to us having to be able
-to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).
-
-"The success of the CG method":"https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf"
-for finding solutions of non-linear problems is based on the theory
-of conjugate gradients for linear systems of equations. It belongs to
-the class of iterative methods for solving problems from linear
-algebra of the type
-!bt
-\begin{equation*}
-\bm{A}\bm{x} = \bm{b}.
-\end{equation*}
-!et
-
-In the iterative process we end up with a problem like
-
-!bt
-\begin{equation*}
- \bm{r}= \bm{b}-\bm{A}\bm{x},
-\end{equation*}
-!et
-where $\bm{r}$ is the so-called residual or error in the iterative process.
-
-When we have found the exact solution, $\bm{r}=0$.
-
-!split
-===== Gradient method =====
-
-The residual is zero when we reach the minimum of the quadratic equation
-!bt
-\begin{equation*}
- P(\bm{x})=\frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T\bm{b},
-\end{equation*}
-!et
-
-with the constraint that the matrix $\bm{A}$ is positive definite and
-symmetric. This defines also the Hessian and we want it to be positive definite.
-
-
-!split
-===== Steepest descent method =====
-
-We denote the initial guess for $\bm{x}$ as $\bm{x}_0$.
-We can assume without loss of generality that
-!bt
-\begin{equation*}
-\bm{x}_0=0,
-\end{equation*}
-!et
-or consider the system
-!bt
-\begin{equation*}
-\bm{A}\bm{z} = \bm{b}-\bm{A}\bm{x}_0,
-\end{equation*}
-!et
-instead.
-
-
-!split
-===== Steepest descent method =====
-!bblock
-One can show that the solution $\bm{x}$ is also the unique minimizer of the quadratic form
-!bt
-\begin{equation*}
- f(\bm{x}) = \frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T \bm{x} , \quad \bm{x}\in\mathbf{R}^n.
-\end{equation*}
-!et
-This suggests taking the first basis vector $\bm{r}_1$ (see below for definition)
-to be the gradient of $f$ at $\bm{x}=\bm{x}_0$,
-which equals
-!bt
-\begin{equation*}
-\bm{A}\bm{x}_0-\bm{b},
-\end{equation*}
-!et
-and
-$\bm{x}_0=0$ it is equal $-\bm{b}$.
-
-!eblock
-
-!split
-===== Final expressions =====
-!bblock
-We can compute the residual iteratively as
-!bt
-\begin{equation*}
-\bm{r}_{k+1}=\bm{b}-\bm{A}\bm{x}_{k+1},
- \end{equation*}
-!et
-which equals
-!bt
-\begin{equation*}
-\bm{b}-\bm{A}(\bm{x}_k+\alpha_k\bm{r}_k),
- \end{equation*}
-!et
-or
-!bt
-\begin{equation*}
-(\bm{b}-\bm{A}\bm{x}_k)-\alpha_k\bm{A}\bm{r}_k,
- \end{equation*}
-!et
-which gives
-
-!bt
-\[
-\alpha_k = \frac{\bm{r}_k^T\bm{r}_k}{\bm{r}_k^T\bm{A}\bm{r}_k}
-\]
-!et
-leading to the iterative scheme
-!bt
-\begin{equation*}
-\bm{x}_{k+1}=\bm{x}_k-\alpha_k\bm{r}_{k},
- \end{equation*}
-!et
-!eblock
-
-
-
-!split
-===== Steepest descent example =====
-
-!bc pycod
-import numpy as np
-import numpy.linalg as la
-
-import scipy.optimize as sopt
-
-import matplotlib.pyplot as pt
-from mpl_toolkits.mplot3d import axes3d
-
-def f(x):
- return x[0]**2 + 3.0*x[1]**2
-
-def df(x):
- return np.array([2*x[0], 6*x[1]])
-
-fig = pt.figure()
-ax = fig.gca(projection="3d")
-
-xmesh, ymesh = np.mgrid[-3:3:50j,-3:3:50j]
-fmesh = f(np.array([xmesh, ymesh]))
-ax.plot_surface(xmesh, ymesh, fmesh)
-!ec
-And then as countor plot
-!bc pycod
-pt.axis("equal")
-pt.contour(xmesh, ymesh, fmesh)
-guesses = [np.array([2, 2./5])]
-!ec
-Find guesses
-!bc pycod
-x = guesses[-1]
-s = -df(x)
-!ec
-Run it!
-!bc pycod
-def f1d(alpha):
- return f(x + alpha*s)
-
-alpha_opt = sopt.golden(f1d)
-next_guess = x + alpha_opt * s
-guesses.append(next_guess)
-print(next_guess)
-!ec
-What happened?
-!bc pycod
-pt.axis("equal")
-pt.contour(xmesh, ymesh, fmesh, 50)
-it_array = np.array(guesses)
-pt.plot(it_array.T[0], it_array.T[1], "x-")
-!ec
-
-Note that we did only one iteration here. We can easily add more using our previous guesses.
-
-!split
-===== Conjugate gradient method =====
-!bblock
-In the CG method we define so-called conjugate directions and two vectors
-$\bm{s}$ and $\bm{t}$
-are said to be
-conjugate if
-!bt
-\begin{equation*}
-\bm{s}^T\bm{A}\bm{t}= 0.
-\end{equation*}
-!et
-The philosophy of the CG method is to perform searches in various conjugate directions
-of our vectors $\bm{x}_i$ obeying the above criterion, namely
-!bt
-\begin{equation*}
-\bm{x}_i^T\bm{A}\bm{x}_j= 0.
-\end{equation*}
-!et
-Two vectors are conjugate if they are orthogonal with respect to
-this inner product. Being conjugate is a symmetric relation: if $\bm{s}$ is conjugate to $\bm{t}$, then $\bm{t}$ is conjugate to $\bm{s}$.
-!eblock
-
-!split
-===== Conjugate gradient method =====
-!bblock
-An example is given by the eigenvectors of the matrix
-!bt
-\begin{equation*}
-\bm{v}_i^T\bm{A}\bm{v}_j= \lambda\bm{v}_i^T\bm{v}_j,
-\end{equation*}
-!et
-which is zero unless $i=j$.
-!eblock
-
-
-!split
-===== Conjugate gradient method =====
-!bblock
-Assume now that we have a symmetric positive-definite matrix $\bm{A}$ of size
-$n\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector
-!bt
-\begin{equation*}
-\bm{x}_{i+1}=\bm{x}_{i}+\alpha_i\bm{p}_{i}.
-\end{equation*}
-!et
-We assume that $\bm{p}_{i}$ is a sequence of $n$ mutually conjugate directions.
-Then the $\bm{p}_{i}$ form a basis of $R^n$ and we can expand the solution
-$ \bm{A}\bm{x} = \bm{b}$ in this basis, namely
-
-!bt
-\begin{equation*}
- \bm{x} = \sum^{n}_{i=1} \alpha_i \bm{p}_i.
-\end{equation*}
-!et
-!eblock
-
-!split
-===== Conjugate gradient method =====
-!bblock
-The coefficients are given by
-!bt
-\begin{equation*}
- \mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}.
-\end{equation*}
-!et
-Multiplying with $\bm{p}_k^T$ from the left gives
-
-!bt
-\begin{equation*}
- \bm{p}_k^T \bm{A}\bm{x} = \sum^{n}_{i=1} \alpha_i\bm{p}_k^T \bm{A}\bm{p}_i= \bm{p}_k^T \bm{b},
-\end{equation*}
-!et
-and we can define the coefficients $\alpha_k$ as
-
-!bt
-\begin{equation*}
- \alpha_k = \frac{\bm{p}_k^T \bm{b}}{\bm{p}_k^T \bm{A} \bm{p}_k}
-\end{equation*}
-!et
-!eblock
-
-!split
-===== Conjugate gradient method and iterations =====
-!bblock
-
-If we choose the conjugate vectors $\bm{p}_k$ carefully,
-then we may not need all of them to obtain a good approximation to the solution
-$\bm{x}$.
-We want to regard the conjugate gradient method as an iterative method.
-This will us to solve systems where $n$ is so large that the direct
-method would take too much time.
-
-We denote the initial guess for $\bm{x}$ as $\bm{x}_0$.
-We can assume without loss of generality that
-!bt
-\begin{equation*}
-\bm{x}_0=0,
-\end{equation*}
-!et
-or consider the system
-!bt
-\begin{equation*}
-\bm{A}\bm{z} = \bm{b}-\bm{A}\bm{x}_0,
-\end{equation*}
-!et
-instead.
-!eblock
-
-
-!split
-===== Conjugate gradient method =====
-!bblock
-One can show that the solution $\bm{x}$ is also the unique minimizer of the quadratic form
-!bt
-\begin{equation*}
- f(\bm{x}) = \frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T \bm{x} , \quad \bm{x}\in\mathbf{R}^n.
-\end{equation*}
-!et
-This suggests taking the first basis vector $\bm{p}_1$
-to be the gradient of $f$ at $\bm{x}=\bm{x}_0$,
-which equals
-!bt
-\begin{equation*}
-\bm{A}\bm{x}_0-\bm{b},
-\end{equation*}
-!et
-and
-$\bm{x}_0=0$ it is equal $-\bm{b}$.
-The other vectors in the basis will be conjugate to the gradient,
-hence the name conjugate gradient method.
-!eblock
-
-
-!split
-===== Conjugate gradient method =====
-!bblock
-Let $\bm{r}_k$ be the residual at the $k$-th step:
-!bt
-\begin{equation*}
-\bm{r}_k=\bm{b}-\bm{A}\bm{x}_k.
-\end{equation*}
-!et
-Note that $\bm{r}_k$ is the negative gradient of $f$ at
-$\bm{x}=\bm{x}_k$,
-so the gradient descent method would be to move in the direction $\bm{r}_k$.
-Here, we insist that the directions $\bm{p}_k$ are conjugate to each other,
-so we take the direction closest to the gradient $\bm{r}_k$
-under the conjugacy constraint.
-This gives the following expression
-!bt
-\begin{equation*}
-\bm{p}_{k+1}=\bm{r}_k-\frac{\bm{p}_k^T \bm{A}\bm{r}_k}{\bm{p}_k^T\bm{A}\bm{p}_k} \bm{p}_k.
-\end{equation*}
-!et
-!eblock
-
-!split
-===== Conjugate gradient method =====
-!bblock
-We can also compute the residual iteratively as
-!bt
-\begin{equation*}
-\bm{r}_{k+1}=\bm{b}-\bm{A}\bm{x}_{k+1},
- \end{equation*}
-!et
-which equals
-!bt
-\begin{equation*}
-\bm{b}-\bm{A}(\bm{x}_k+\alpha_k\bm{p}_k),
- \end{equation*}
-!et
-or
-!bt
-\begin{equation*}
-(\bm{b}-\bm{A}\bm{x}_k)-\alpha_k\bm{A}\bm{p}_k,
- \end{equation*}
-!et
-which gives
-
-!bt
-\begin{equation*}
-\bm{r}_{k+1}=\bm{r}_k-\bm{A}\bm{p}_{k},
- \end{equation*}
-!et
-!eblock
-
-
-
!split