From f7578d8e14287b65ecd3232ba2d27b8188efcfad Mon Sep 17 00:00:00 2001 From: mhjensen Date: Fri, 25 Sep 2020 05:19:28 +0200 Subject: [PATCH] replacing hats with boldface --- doc/pub/week39/html/._week39-bs000.html | 2 +- doc/pub/week39/html/._week39-bs004.html | 6 +- doc/pub/week39/html/._week39-bs005.html | 16 +- doc/pub/week39/html/._week39-bs006.html | 4 +- doc/pub/week39/html/._week39-bs010.html | 8 +- doc/pub/week39/html/._week39-bs020.html | 10 +- doc/pub/week39/html/._week39-bs021.html | 4 +- doc/pub/week39/html/._week39-bs022.html | 6 +- doc/pub/week39/html/._week39-bs023.html | 12 +- doc/pub/week39/html/._week39-bs024.html | 10 +- doc/pub/week39/html/._week39-bs026.html | 10 +- doc/pub/week39/html/._week39-bs027.html | 2 +- doc/pub/week39/html/._week39-bs028.html | 12 +- doc/pub/week39/html/._week39-bs029.html | 6 +- doc/pub/week39/html/._week39-bs030.html | 10 +- doc/pub/week39/html/._week39-bs031.html | 12 +- doc/pub/week39/html/._week39-bs032.html | 16 +- doc/pub/week39/html/._week39-bs033.html | 8 +- doc/pub/week39/html/._week39-bs034.html | 4 +- doc/pub/week39/html/._week39-bs035.html | 4 +- doc/pub/week39/html/._week39-bs037.html | 2 +- doc/pub/week39/html/._week39-bs058.html | 12 +- doc/pub/week39/html/week39-bs.html | 2 +- doc/pub/week39/html/week39-reveal.html | 176 +++++++++---------- doc/pub/week39/html/week39-solarized.html | 176 +++++++++---------- doc/pub/week39/html/week39.html | 176 +++++++++---------- doc/pub/week39/ipynb/ipynb-week39-src.tar.gz | Bin 191 -> 192 bytes doc/pub/week39/ipynb/week39.ipynb | 176 +++++++++---------- doc/src/week39/week39.do.txt | 174 +++++++++--------- 29 files changed, 528 insertions(+), 528 deletions(-) diff --git a/doc/pub/week39/html/._week39-bs000.html b/doc/pub/week39/html/._week39-bs000.html index 2e9c5fff5..86bba2bb1 100644 --- a/doc/pub/week39/html/._week39-bs000.html +++ b/doc/pub/week39/html/._week39-bs000.html @@ -278,7 +278,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 24, 2020

+

Sep 25, 2020


diff --git a/doc/pub/week39/html/._week39-bs004.html b/doc/pub/week39/html/._week39-bs004.html index 8d735bf5b..615b42d76 100644 --- a/doc/pub/week39/html/._week39-bs004.html +++ b/doc/pub/week39/html/._week39-bs004.html @@ -271,12 +271,12 @@ defined probabilities $$ \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), \end{align*} $$ -where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). +where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).

diff --git a/doc/pub/week39/html/._week39-bs005.html b/doc/pub/week39/html/._week39-bs005.html index e3dcc869f..7757f933c 100644 --- a/doc/pub/week39/html/._week39-bs005.html +++ b/doc/pub/week39/html/._week39-bs005.html @@ -262,22 +262,22 @@ MathJax.Hub.Config({

The equations to solve

-Our compact equations used a definition of a vector \( \hat{y} \) with \( n \) -elements \( y_i \), an \( n\times p \) matrix \( \hat{X} \) which contains the -\( x_i \) values and a vector \( \hat{p} \) of fitted probabilities -\( p(y_i\vert x_i,\hat{\beta}) \). We rewrote in a more compact form +Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) +elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the +\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities +\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form the first derivative of the cost function as $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$

-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as $$ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$ This defines what is called the Hessian matrix. diff --git a/doc/pub/week39/html/._week39-bs006.html b/doc/pub/week39/html/._week39-bs006.html index a6b5d20ea..c8aeb0133 100644 --- a/doc/pub/week39/html/._week39-bs006.html +++ b/doc/pub/week39/html/._week39-bs006.html @@ -268,13 +268,13 @@ If we can set up these equations, Newton-Raphson's iterative method is normally Our iterative scheme is then given by $$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T}\right)^{-1}_{\hat{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}}\right)_{\hat{\beta}^{\mathrm{old}}}, +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, $$ or in matrix form as $$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\hat{X}^T\hat{W}\hat{X} \right)^{-1}\times \left(-\hat{X}^T(\hat{y}-\hat{p}) \right)_{\hat{\beta}^{\mathrm{old}}}. +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. $$ The right-hand side is computed with the old values of \( \beta \). diff --git a/doc/pub/week39/html/._week39-bs010.html b/doc/pub/week39/html/._week39-bs010.html index 5adfecdc0..1319947cd 100644 --- a/doc/pub/week39/html/._week39-bs010.html +++ b/doc/pub/week39/html/._week39-bs010.html @@ -281,9 +281,9 @@ $$ \end{array}. $$ -Defining the Jacobian matrix \( {\bf \hat{J}} \) we have +Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have $$ - {\bf \hat{J}}=\left( \begin{array}{cc} + {\bf \boldsymbol{J}}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ \partial f_2/\partial x_1 &\partial f_2/\partial x_2 \end{array} \right), @@ -299,13 +299,13 @@ $$ where we have defined $$ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= - -{\bf \hat{J}}^{-1} + -{\bf \boldsymbol{J}}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). $$ We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may -arise in case \( {\bf \hat{J}} \) is nearly singular. +arise in case \( {\bf \boldsymbol{J}} \) is nearly singular.

It is rather straightforward to extend the above scheme to systems of diff --git a/doc/pub/week39/html/._week39-bs020.html b/doc/pub/week39/html/._week39-bs020.html index 394900c6e..49bb20c2e 100644 --- a/doc/pub/week39/html/._week39-bs020.html +++ b/doc/pub/week39/html/._week39-bs020.html @@ -263,7 +263,7 @@ MathJax.Hub.Config({

Before we proceed, we would like to discuss the approach called the -standard Steepest descent, which again leads to us having to be able +standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).

@@ -274,7 +274,7 @@ the class of iterative methods for solving problems from linear algebra of the type $$ \begin{equation*} -\hat{A}\hat{x} = \hat{b}. +\boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}. \end{equation*} $$ @@ -283,14 +283,14 @@ In the iterative process we end up with a problem like $$ \begin{equation*} - \hat{r}= \hat{b}-\hat{A}\hat{x}, + \boldsymbol{r}= \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}, \end{equation*} $$ -where \( \hat{r} \) is the so-called residual or error in the iterative process. +where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.

-When we have found the exact solution, \( \hat{r}=0 \). +When we have found the exact solution, \( \boldsymbol{r}=0 \).

diff --git a/doc/pub/week39/html/._week39-bs021.html b/doc/pub/week39/html/._week39-bs021.html index ed02517e3..e9bb35144 100644 --- a/doc/pub/week39/html/._week39-bs021.html +++ b/doc/pub/week39/html/._week39-bs021.html @@ -265,12 +265,12 @@ MathJax.Hub.Config({ The residual is zero when we reach the minimum of the quadratic equation $$ \begin{equation*} - P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b}, + P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b}, \end{equation*} $$

-with the constraint that the matrix \( \hat{A} \) is positive definite and +with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and symmetric. This defines also the Hessian and we want it to be positive definite.

diff --git a/doc/pub/week39/html/._week39-bs022.html b/doc/pub/week39/html/._week39-bs022.html index 0e7fda90a..388f6c125 100644 --- a/doc/pub/week39/html/._week39-bs022.html +++ b/doc/pub/week39/html/._week39-bs022.html @@ -262,18 +262,18 @@ MathJax.Hub.Config({

Steepest descent method

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that $$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$ or consider the system $$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ diff --git a/doc/pub/week39/html/._week39-bs023.html b/doc/pub/week39/html/._week39-bs023.html index 21711e289..753907b3c 100644 --- a/doc/pub/week39/html/._week39-bs023.html +++ b/doc/pub/week39/html/._week39-bs023.html @@ -263,24 +263,24 @@ MathJax.Hub.Config({

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form $$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \hat{r}_1 \) (see below for definition) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals $$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).

diff --git a/doc/pub/week39/html/._week39-bs024.html b/doc/pub/week39/html/._week39-bs024.html index 37e4be7cd..7853e9cd0 100644 --- a/doc/pub/week39/html/._week39-bs024.html +++ b/doc/pub/week39/html/._week39-bs024.html @@ -266,34 +266,34 @@ MathJax.Hub.Config({ We can compute the residual iteratively as $$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ which equals $$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{r}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k), \end{equation*} $$ or $$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{r}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k, \end{equation*} $$ which gives $$ -\alpha_k = \frac{\hat{r}_k^T\hat{r}_k}{\hat{r}_k^T\hat{A}\hat{r}_k} +\alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k} $$ leading to the iterative scheme $$ \begin{equation*} -\hat{x}_{k+1}=\hat{x}_k-\alpha_k\hat{r}_{k}, +\boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k}, \end{equation*} $$
diff --git a/doc/pub/week39/html/._week39-bs026.html b/doc/pub/week39/html/._week39-bs026.html index 706f78a27..a9cc6e277 100644 --- a/doc/pub/week39/html/._week39-bs026.html +++ b/doc/pub/week39/html/._week39-bs026.html @@ -264,25 +264,25 @@ MathJax.Hub.Config({

In the CG method we define so-called conjugate directions and two vectors -\( \hat{s} \) and \( \hat{t} \) +\( \boldsymbol{s} \) and \( \boldsymbol{t} \) are said to be conjugate if $$ \begin{equation*} -\hat{s}^T\hat{A}\hat{t}= 0. +\boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0. \end{equation*} $$ The philosophy of the CG method is to perform searches in various conjugate directions -of our vectors \( \hat{x}_i \) obeying the above criterion, namely +of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely $$ \begin{equation*} -\hat{x}_i^T\hat{A}\hat{x}_j= 0. +\boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0. \end{equation*} $$ Two vectors are conjugate if they are orthogonal with respect to -this inner product. Being conjugate is a symmetric relation: if \( \hat{s} \) is conjugate to \( \hat{t} \), then \( \hat{t} \) is conjugate to \( \hat{s} \). +this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \).

diff --git a/doc/pub/week39/html/._week39-bs027.html b/doc/pub/week39/html/._week39-bs027.html index 0c99ee4be..79b111979 100644 --- a/doc/pub/week39/html/._week39-bs027.html +++ b/doc/pub/week39/html/._week39-bs027.html @@ -266,7 +266,7 @@ MathJax.Hub.Config({ An example is given by the eigenvectors of the matrix $$ \begin{equation*} -\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j, +\boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j, \end{equation*} $$ diff --git a/doc/pub/week39/html/._week39-bs028.html b/doc/pub/week39/html/._week39-bs028.html index 30af301c5..71f50f6e8 100644 --- a/doc/pub/week39/html/._week39-bs028.html +++ b/doc/pub/week39/html/._week39-bs028.html @@ -263,21 +263,21 @@ MathJax.Hub.Config({

-Assume now that we have a symmetric positive-definite matrix \( \hat{A} \) of size +Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size \( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector $$ \begin{equation*} -\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}. +\boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}. \end{equation*} $$ -We assume that \( \hat{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. -Then the \( \hat{p}_{i} \) form a basis of \( R^n \) and we can expand the solution -$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely +We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. +Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution +$ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely $$ \begin{equation*} - \hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i. + \boldsymbol{x} = \sum^{n}_{i=1} \alpha_i \boldsymbol{p}_i. \end{equation*} $$

diff --git a/doc/pub/week39/html/._week39-bs029.html b/doc/pub/week39/html/._week39-bs029.html index 8ac2ba899..893f9bd01 100644 --- a/doc/pub/week39/html/._week39-bs029.html +++ b/doc/pub/week39/html/._week39-bs029.html @@ -270,11 +270,11 @@ $$ \end{equation*} $$ -Multiplying with \( \hat{p}_k^T \) from the left gives +Multiplying with \( \boldsymbol{p}_k^T \) from the left gives $$ \begin{equation*} - \hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b}, + \boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{x} = \sum^{n}_{i=1} \alpha_i\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{p}_i= \boldsymbol{p}_k^T \boldsymbol{b}, \end{equation*} $$ @@ -282,7 +282,7 @@ and we can define the coefficients \( \alpha_k \) as $$ \begin{equation*} - \alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k} + \alpha_k = \frac{\boldsymbol{p}_k^T \boldsymbol{b}}{\boldsymbol{p}_k^T \boldsymbol{A} \boldsymbol{p}_k} \end{equation*} $$
diff --git a/doc/pub/week39/html/._week39-bs030.html b/doc/pub/week39/html/._week39-bs030.html index cfcb8b8d4..1462b9142 100644 --- a/doc/pub/week39/html/._week39-bs030.html +++ b/doc/pub/week39/html/._week39-bs030.html @@ -265,26 +265,26 @@ MathJax.Hub.Config({

-If we choose the conjugate vectors \( \hat{p}_k \) carefully, +If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, then we may not need all of them to obtain a good approximation to the solution -\( \hat{x} \). +\( \boldsymbol{x} \). We want to regard the conjugate gradient method as an iterative method. This will us to solve systems where \( n \) is so large that the direct method would take too much time.

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that $$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$ or consider the system $$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ diff --git a/doc/pub/week39/html/._week39-bs031.html b/doc/pub/week39/html/._week39-bs031.html index ec79c4982..4b7e7ca05 100644 --- a/doc/pub/week39/html/._week39-bs031.html +++ b/doc/pub/week39/html/._week39-bs031.html @@ -263,24 +263,24 @@ MathJax.Hub.Config({

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form $$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \hat{p}_1 \) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{p}_1 \) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals $$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method.

diff --git a/doc/pub/week39/html/._week39-bs032.html b/doc/pub/week39/html/._week39-bs032.html index 282ae88ab..20c15d529 100644 --- a/doc/pub/week39/html/._week39-bs032.html +++ b/doc/pub/week39/html/._week39-bs032.html @@ -263,23 +263,23 @@ MathJax.Hub.Config({

-Let \( \hat{r}_k \) be the residual at the \( k \)-th step: +Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step: $$ \begin{equation*} -\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k. +\boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k. \end{equation*} $$ -Note that \( \hat{r}_k \) is the negative gradient of \( f \) at -\( \hat{x}=\hat{x}_k \), -so the gradient descent method would be to move in the direction \( \hat{r}_k \). -Here, we insist that the directions \( \hat{p}_k \) are conjugate to each other, -so we take the direction closest to the gradient \( \hat{r}_k \) +Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at +\( \boldsymbol{x}=\boldsymbol{x}_k \), +so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \). +Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other, +so we take the direction closest to the gradient \( \boldsymbol{r}_k \) under the conjugacy constraint. This gives the following expression $$ \begin{equation*} -\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k. +\boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k. \end{equation*} $$

diff --git a/doc/pub/week39/html/._week39-bs033.html b/doc/pub/week39/html/._week39-bs033.html index 4eb0da943..4f3978e77 100644 --- a/doc/pub/week39/html/._week39-bs033.html +++ b/doc/pub/week39/html/._week39-bs033.html @@ -266,21 +266,21 @@ MathJax.Hub.Config({ We can also compute the residual iteratively as $$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ which equals $$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k), \end{equation*} $$ or $$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k, \end{equation*} $$ @@ -288,7 +288,7 @@ which gives $$ \begin{equation*} -\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k}, +\boldsymbol{r}_{k+1}=\boldsymbol{r}_k-\boldsymbol{A}\boldsymbol{p}_{k}, \end{equation*} $$
diff --git a/doc/pub/week39/html/._week39-bs034.html b/doc/pub/week39/html/._week39-bs034.html index 7f28fb2cb..1e05ee655 100644 --- a/doc/pub/week39/html/._week39-bs034.html +++ b/doc/pub/week39/html/._week39-bs034.html @@ -281,12 +281,12 @@ $$ with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by $$ -h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, +h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, $$ such that $$ -\hat{y}_i = \beta_0 + \beta_1 x_i. +\boldsymbol{y}_i = \beta_0 + \beta_1 x_i. $$

diff --git a/doc/pub/week39/html/._week39-bs035.html b/doc/pub/week39/html/._week39-bs035.html index fbeabf47a..e112a8dac 100644 --- a/doc/pub/week39/html/._week39-bs035.html +++ b/doc/pub/week39/html/._week39-bs035.html @@ -262,10 +262,10 @@ MathJax.Hub.Config({

Gradient descent example

-Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) +Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

-It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by $$ X \equiv \begin{bmatrix} 1 &; x_1 \\ diff --git a/doc/pub/week39/html/._week39-bs037.html b/doc/pub/week39/html/._week39-bs037.html index f64a6c2dc..112fed356 100644 --- a/doc/pub/week39/html/._week39-bs037.html +++ b/doc/pub/week39/html/._week39-bs037.html @@ -262,7 +262,7 @@ MathJax.Hub.Config({

The Hessian matrix

The Hessian matrix of \( C(\beta) \) is given by $$ -\hat{H} \equiv \begin{bmatrix} +\boldsymbol{H} \equiv \begin{bmatrix} \frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ \end{bmatrix} = 2X^T X. diff --git a/doc/pub/week39/html/._week39-bs058.html b/doc/pub/week39/html/._week39-bs058.html index d3c59b373..c47b49518 100644 --- a/doc/pub/week39/html/._week39-bs058.html +++ b/doc/pub/week39/html/._week39-bs058.html @@ -281,9 +281,9 @@ $$ \tag{5}\\ \mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ \mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ -\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \tag{6} \end{align} $$ @@ -297,12 +297,12 @@ respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance -\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - -(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - +(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by $$ -\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. +\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$

diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 2e9c5fff5..86bba2bb1 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -278,7 +278,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 24, 2020

+

Sep 25, 2020


diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index b973ae8ac..6ae6497c6 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

 
-

Sep 24, 2020

+

Sep 25, 2020


@@ -208,13 +208,13 @@ defined probabilities

 
$$ \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), \end{align*} $$

 
-where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). +where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). @@ -222,25 +222,25 @@ where \( \hat{\beta} \) are the weights we wish to extract from data, in our cas

The equations to solve

-Our compact equations used a definition of a vector \( \hat{y} \) with \( n \) -elements \( y_i \), an \( n\times p \) matrix \( \hat{X} \) which contains the -\( x_i \) values and a vector \( \hat{p} \) of fitted probabilities -\( p(y_i\vert x_i,\hat{\beta}) \). We rewrote in a more compact form +Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) +elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the +\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities +\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form the first derivative of the cost function as

 
$$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$

 

-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as

 
$$ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$

 
@@ -259,7 +259,7 @@ Our iterative scheme is then given by

 
$$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T}\right)^{-1}_{\hat{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}}\right)_{\hat{\beta}^{\mathrm{old}}}, +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, $$

 
@@ -267,7 +267,7 @@ or in matrix form as

 
$$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\hat{X}^T\hat{W}\hat{X} \right)^{-1}\times \left(-\hat{X}^T(\hat{y}-\hat{p}) \right)_{\hat{\beta}^{\mathrm{old}}}. +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. $$

 
@@ -384,10 +384,10 @@ $$ $$

 
-Defining the Jacobian matrix \( {\bf \hat{J}} \) we have +Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have

 
$$ - {\bf \hat{J}}=\left( \begin{array}{cc} + {\bf \boldsymbol{J}}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ \partial f_2/\partial x_1 &\partial f_2/\partial x_2 \end{array} \right), @@ -407,14 +407,14 @@ where we have defined

 
$$ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= - -{\bf \hat{J}}^{-1} + -{\bf \boldsymbol{J}}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). $$

 
We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may -arise in case \( {\bf \hat{J}} \) is nearly singular. +arise in case \( {\bf \boldsymbol{J}} \) is nearly singular.

It is rather straightforward to extend the above scheme to systems of @@ -647,7 +647,7 @@ Using the definition of convexity, try to show that a function satisfying the pr

Before we proceed, we would like to discuss the approach called the -standard Steepest descent, which again leads to us having to be able +standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).

@@ -659,7 +659,7 @@ algebra of the type

 
$$ \begin{equation*} -\hat{A}\hat{x} = \hat{b}. +\boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}. \end{equation*} $$

 
@@ -670,15 +670,15 @@ In the iterative process we end up with a problem like

 
$$ \begin{equation*} - \hat{r}= \hat{b}-\hat{A}\hat{x}, + \boldsymbol{r}= \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}, \end{equation*} $$

 
-where \( \hat{r} \) is the so-called residual or error in the iterative process. +where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.

-When we have found the exact solution, \( \hat{r}=0 \). +When we have found the exact solution, \( \boldsymbol{r}=0 \). @@ -690,13 +690,13 @@ The residual is zero when we reach the minimum of the quadratic equation

 
$$ \begin{equation*} - P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b}, + P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b}, \end{equation*} $$

 

-with the constraint that the matrix \( \hat{A} \) is positive definite and +with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and symmetric. This defines also the Hessian and we want it to be positive definite. @@ -705,12 +705,12 @@ symmetric. This defines also the Hessian and we want it to be positive definit

Steepest descent method

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that

 
$$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$

 
@@ -719,7 +719,7 @@ or consider the system

 
$$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$

 
@@ -733,28 +733,28 @@ instead.

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form

 
$$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$

 
-This suggests taking the first basis vector \( \hat{r}_1 \) (see below for definition) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals

 
$$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$

 
and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).

@@ -770,7 +770,7 @@ We can compute the residual iteratively as

 
$$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$

 
@@ -779,7 +779,7 @@ which equals

 
$$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{r}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k), \end{equation*} $$

 
@@ -788,7 +788,7 @@ or

 
$$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{r}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k, \end{equation*} $$

 
@@ -797,7 +797,7 @@ which gives

 
$$ -\alpha_k = \frac{\hat{r}_k^T\hat{r}_k}{\hat{r}_k^T\hat{A}\hat{r}_k} +\alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k} $$

 
@@ -805,7 +805,7 @@ leading to the iterative scheme

 
$$ \begin{equation*} -\hat{x}_{k+1}=\hat{x}_k-\alpha_k\hat{r}_{k}, +\boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k}, \end{equation*} $$

 
@@ -889,29 +889,29 @@ pt.plot(it_array.T[0], it_array.T[

In the CG method we define so-called conjugate directions and two vectors -\( \hat{s} \) and \( \hat{t} \) +\( \boldsymbol{s} \) and \( \boldsymbol{t} \) are said to be conjugate if

 
$$ \begin{equation*} -\hat{s}^T\hat{A}\hat{t}= 0. +\boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0. \end{equation*} $$

 
The philosophy of the CG method is to perform searches in various conjugate directions -of our vectors \( \hat{x}_i \) obeying the above criterion, namely +of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely

 
$$ \begin{equation*} -\hat{x}_i^T\hat{A}\hat{x}_j= 0. +\boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0. \end{equation*} $$

 
Two vectors are conjugate if they are orthogonal with respect to -this inner product. Being conjugate is a symmetric relation: if \( \hat{s} \) is conjugate to \( \hat{t} \), then \( \hat{t} \) is conjugate to \( \hat{s} \). +this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \).

@@ -925,7 +925,7 @@ An example is given by the eigenvectors of the matrix

 
$$ \begin{equation*} -\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j, +\boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j, \end{equation*} $$

 
@@ -940,24 +940,24 @@ which is zero unless \( i=j \).

-Assume now that we have a symmetric positive-definite matrix \( \hat{A} \) of size +Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size \( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector

 
$$ \begin{equation*} -\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}. +\boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}. \end{equation*} $$

 
-We assume that \( \hat{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. -Then the \( \hat{p}_{i} \) form a basis of \( R^n \) and we can expand the solution -$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely +We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. +Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution +$ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely

 
$$ \begin{equation*} - \hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i. + \boldsymbol{x} = \sum^{n}_{i=1} \alpha_i \boldsymbol{p}_i. \end{equation*} $$

 
@@ -979,12 +979,12 @@ $$ $$

 
-Multiplying with \( \hat{p}_k^T \) from the left gives +Multiplying with \( \boldsymbol{p}_k^T \) from the left gives

 
$$ \begin{equation*} - \hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b}, + \boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{x} = \sum^{n}_{i=1} \alpha_i\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{p}_i= \boldsymbol{p}_k^T \boldsymbol{b}, \end{equation*} $$

 
@@ -994,7 +994,7 @@ and we can define the coefficients \( \alpha_k \) as

 
$$ \begin{equation*} - \alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k} + \alpha_k = \frac{\boldsymbol{p}_k^T \boldsymbol{b}}{\boldsymbol{p}_k^T \boldsymbol{A} \boldsymbol{p}_k} \end{equation*} $$

 
@@ -1007,20 +1007,20 @@ $$

-If we choose the conjugate vectors \( \hat{p}_k \) carefully, +If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, then we may not need all of them to obtain a good approximation to the solution -\( \hat{x} \). +\( \boldsymbol{x} \). We want to regard the conjugate gradient method as an iterative method. This will us to solve systems where \( n \) is so large that the direct method would take too much time.

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that

 
$$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$

 
@@ -1029,7 +1029,7 @@ or consider the system

 
$$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$

 
@@ -1044,28 +1044,28 @@ instead.

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form

 
$$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$

 
-This suggests taking the first basis vector \( \hat{p}_1 \) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{p}_1 \) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals

 
$$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$

 
and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method.

@@ -1077,26 +1077,26 @@ hence the name conjugate gradient method.

-Let \( \hat{r}_k \) be the residual at the \( k \)-th step: +Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step:

 
$$ \begin{equation*} -\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k. +\boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k. \end{equation*} $$

 
-Note that \( \hat{r}_k \) is the negative gradient of \( f \) at -\( \hat{x}=\hat{x}_k \), -so the gradient descent method would be to move in the direction \( \hat{r}_k \). -Here, we insist that the directions \( \hat{p}_k \) are conjugate to each other, -so we take the direction closest to the gradient \( \hat{r}_k \) +Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at +\( \boldsymbol{x}=\boldsymbol{x}_k \), +so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \). +Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other, +so we take the direction closest to the gradient \( \boldsymbol{r}_k \) under the conjugacy constraint. This gives the following expression

 
$$ \begin{equation*} -\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k. +\boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k. \end{equation*} $$

 
@@ -1113,7 +1113,7 @@ We can also compute the residual iteratively as

 
$$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$

 
@@ -1122,7 +1122,7 @@ which equals

 
$$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k), \end{equation*} $$

 
@@ -1131,7 +1131,7 @@ or

 
$$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k, \end{equation*} $$

 
@@ -1141,7 +1141,7 @@ which gives

 
$$ \begin{equation*} -\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k}, +\boldsymbol{r}_{k+1}=\boldsymbol{r}_k-\boldsymbol{A}\boldsymbol{p}_{k}, \end{equation*} $$

 
@@ -1176,14 +1176,14 @@ with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additiona The linear regression model is given by

 
$$ -h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, +h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, $$

 
such that

 
$$ -\hat{y}_i = \beta_0 + \beta_1 x_i. +\boldsymbol{y}_i = \beta_0 + \beta_1 x_i. $$

 
@@ -1193,10 +1193,10 @@ $$

Gradient descent example

-Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) +Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

-It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by

 
$$ X \equiv \begin{bmatrix} @@ -1240,7 +1240,7 @@ where \( X \) is the design matrix defined above. The Hessian matrix of \( C(\beta) \) is given by

 
$$ -\hat{H} \equiv \begin{bmatrix} +\boldsymbol{H} \equiv \begin{bmatrix} \frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ \end{bmatrix} = 2X^T X. @@ -1938,9 +1938,9 @@ $$ \tag{5}\\ \mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ \mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ -\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \tag{6} \end{align} $$ @@ -1955,13 +1955,13 @@ respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance -\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - -(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - +(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by

 
$$ -\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. +\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$

 
diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index 8c85728a9..ee2540461 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -198,7 +198,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 24, 2020

+

Sep 25, 2020












@@ -250,12 +250,12 @@ defined probabilities $$ \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), \end{align*} $$ -where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). +where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).











@@ -263,22 +263,22 @@ where \( \hat{\beta} \) are the weights we wish to extract from data, in our cas

The equations to solve

-Our compact equations used a definition of a vector \( \hat{y} \) with \( n \) -elements \( y_i \), an \( n\times p \) matrix \( \hat{X} \) which contains the -\( x_i \) values and a vector \( \hat{p} \) of fitted probabilities -\( p(y_i\vert x_i,\hat{\beta}) \). We rewrote in a more compact form +Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) +elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the +\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities +\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form the first derivative of the cost function as $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$

-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as $$ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$ This defines what is called the Hessian matrix. @@ -295,13 +295,13 @@ If we can set up these equations, Newton-Raphson's iterative method is normally Our iterative scheme is then given by $$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T}\right)^{-1}_{\hat{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}}\right)_{\hat{\beta}^{\mathrm{old}}}, +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, $$ or in matrix form as $$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\hat{X}^T\hat{W}\hat{X} \right)^{-1}\times \left(-\hat{X}^T(\hat{y}-\hat{p}) \right)_{\hat{\beta}^{\mathrm{old}}}. +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. $$ The right-hand side is computed with the old values of \( \beta \). @@ -405,9 +405,9 @@ $$ \end{array}. $$ -Defining the Jacobian matrix \( {\bf \hat{J}} \) we have +Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have $$ - {\bf \hat{J}}=\left( \begin{array}{cc} + {\bf \boldsymbol{J}}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ \partial f_2/\partial x_1 &\partial f_2/\partial x_2 \end{array} \right), @@ -423,13 +423,13 @@ $$ where we have defined $$ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= - -{\bf \hat{J}}^{-1} + -{\bf \boldsymbol{J}}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). $$ We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may -arise in case \( {\bf \hat{J}} \) is nearly singular. +arise in case \( {\bf \boldsymbol{J}} \) is nearly singular.

It is rather straightforward to extend the above scheme to systems of @@ -657,7 +657,7 @@ Using the definition of convexity, try to show that a function satisfying the pr

Before we proceed, we would like to discuss the approach called the -standard Steepest descent, which again leads to us having to be able +standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).

@@ -668,7 +668,7 @@ the class of iterative methods for solving problems from linear algebra of the type $$ \begin{equation*} -\hat{A}\hat{x} = \hat{b}. +\boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}. \end{equation*} $$ @@ -677,14 +677,14 @@ In the iterative process we end up with a problem like $$ \begin{equation*} - \hat{r}= \hat{b}-\hat{A}\hat{x}, + \boldsymbol{r}= \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}, \end{equation*} $$ -where \( \hat{r} \) is the so-called residual or error in the iterative process. +where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.

-When we have found the exact solution, \( \hat{r}=0 \). +When we have found the exact solution, \( \boldsymbol{r}=0 \).











@@ -695,12 +695,12 @@ When we have found the exact solution, \( \hat{r}=0 \). The residual is zero when we reach the minimum of the quadratic equation $$ \begin{equation*} - P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b}, + P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b}, \end{equation*} $$

-with the constraint that the matrix \( \hat{A} \) is positive definite and +with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and symmetric. This defines also the Hessian and we want it to be positive definite.

@@ -709,18 +709,18 @@ symmetric. This defines also the Hessian and we want it to be positive definit

Steepest descent method

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that $$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$ or consider the system $$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ @@ -733,24 +733,24 @@ instead.

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form $$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \hat{r}_1 \) (see below for definition) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals $$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).

@@ -766,34 +766,34 @@ and We can compute the residual iteratively as $$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ which equals $$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{r}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k), \end{equation*} $$ or $$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{r}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k, \end{equation*} $$ which gives $$ -\alpha_k = \frac{\hat{r}_k^T\hat{r}_k}{\hat{r}_k^T\hat{A}\hat{r}_k} +\alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k} $$ leading to the iterative scheme $$ \begin{equation*} -\hat{x}_{k+1}=\hat{x}_k-\alpha_k\hat{r}_{k}, +\boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k}, \end{equation*} $$
@@ -876,25 +876,25 @@ pt.plot(it_array.T[0], it_array.T[

In the CG method we define so-called conjugate directions and two vectors -\( \hat{s} \) and \( \hat{t} \) +\( \boldsymbol{s} \) and \( \boldsymbol{t} \) are said to be conjugate if $$ \begin{equation*} -\hat{s}^T\hat{A}\hat{t}= 0. +\boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0. \end{equation*} $$ The philosophy of the CG method is to perform searches in various conjugate directions -of our vectors \( \hat{x}_i \) obeying the above criterion, namely +of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely $$ \begin{equation*} -\hat{x}_i^T\hat{A}\hat{x}_j= 0. +\boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0. \end{equation*} $$ Two vectors are conjugate if they are orthogonal with respect to -this inner product. Being conjugate is a symmetric relation: if \( \hat{s} \) is conjugate to \( \hat{t} \), then \( \hat{t} \) is conjugate to \( \hat{s} \). +this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \).

@@ -908,7 +908,7 @@ this inner product. Being conjugate is a symmetric relation: if \( \hat{s} \) is An example is given by the eigenvectors of the matrix $$ \begin{equation*} -\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j, +\boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j, \end{equation*} $$ @@ -923,21 +923,21 @@ which is zero unless \( i=j \).

-Assume now that we have a symmetric positive-definite matrix \( \hat{A} \) of size +Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size \( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector $$ \begin{equation*} -\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}. +\boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}. \end{equation*} $$ -We assume that \( \hat{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. -Then the \( \hat{p}_{i} \) form a basis of \( R^n \) and we can expand the solution -$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely +We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. +Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution +$ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely $$ \begin{equation*} - \hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i. + \boldsymbol{x} = \sum^{n}_{i=1} \alpha_i \boldsymbol{p}_i. \end{equation*} $$

@@ -957,11 +957,11 @@ $$ \end{equation*} $$ -Multiplying with \( \hat{p}_k^T \) from the left gives +Multiplying with \( \boldsymbol{p}_k^T \) from the left gives $$ \begin{equation*} - \hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b}, + \boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{x} = \sum^{n}_{i=1} \alpha_i\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{p}_i= \boldsymbol{p}_k^T \boldsymbol{b}, \end{equation*} $$ @@ -969,7 +969,7 @@ and we can define the coefficients \( \alpha_k \) as $$ \begin{equation*} - \alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k} + \alpha_k = \frac{\boldsymbol{p}_k^T \boldsymbol{b}}{\boldsymbol{p}_k^T \boldsymbol{A} \boldsymbol{p}_k} \end{equation*} $$
@@ -984,26 +984,26 @@ $$

-If we choose the conjugate vectors \( \hat{p}_k \) carefully, +If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, then we may not need all of them to obtain a good approximation to the solution -\( \hat{x} \). +\( \boldsymbol{x} \). We want to regard the conjugate gradient method as an iterative method. This will us to solve systems where \( n \) is so large that the direct method would take too much time.

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that $$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$ or consider the system $$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ @@ -1018,24 +1018,24 @@ instead.

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form $$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \hat{p}_1 \) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{p}_1 \) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals $$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method.

@@ -1048,23 +1048,23 @@ hence the name conjugate gradient method.

-Let \( \hat{r}_k \) be the residual at the \( k \)-th step: +Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step: $$ \begin{equation*} -\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k. +\boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k. \end{equation*} $$ -Note that \( \hat{r}_k \) is the negative gradient of \( f \) at -\( \hat{x}=\hat{x}_k \), -so the gradient descent method would be to move in the direction \( \hat{r}_k \). -Here, we insist that the directions \( \hat{p}_k \) are conjugate to each other, -so we take the direction closest to the gradient \( \hat{r}_k \) +Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at +\( \boldsymbol{x}=\boldsymbol{x}_k \), +so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \). +Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other, +so we take the direction closest to the gradient \( \boldsymbol{r}_k \) under the conjugacy constraint. This gives the following expression $$ \begin{equation*} -\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k. +\boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k. \end{equation*} $$

@@ -1080,21 +1080,21 @@ $$ We can also compute the residual iteratively as $$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ which equals $$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k), \end{equation*} $$ or $$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k, \end{equation*} $$ @@ -1102,7 +1102,7 @@ which gives $$ \begin{equation*} -\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k}, +\boldsymbol{r}_{k+1}=\boldsymbol{r}_k-\boldsymbol{A}\boldsymbol{p}_{k}, \end{equation*} $$ @@ -1133,12 +1133,12 @@ $$ with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by $$ -h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, +h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, $$ such that $$ -\hat{y}_i = \beta_0 + \beta_1 x_i. +\boldsymbol{y}_i = \beta_0 + \beta_1 x_i. $$

@@ -1147,10 +1147,10 @@ $$

Gradient descent example

-Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) +Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

-It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by $$ X \equiv \begin{bmatrix} 1 &; x_1 \\ @@ -1187,7 +1187,7 @@ where \( X \) is the design matrix defined above.

The Hessian matrix

The Hessian matrix of \( C(\beta) \) is given by $$ -\hat{H} \equiv \begin{bmatrix} +\boldsymbol{H} \equiv \begin{bmatrix} \frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ \end{bmatrix} = 2X^T X. @@ -1844,9 +1844,9 @@ $$ \label{_auto4}\\ \mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ \mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ -\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \label{_auto5} \end{align} $$ @@ -1860,12 +1860,12 @@ respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance -\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - -(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - +(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by $$ -\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. +\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$

diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index 45ab68aa8..6ff562ef6 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -203,7 +203,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 24, 2020

+

Sep 25, 2020












@@ -255,12 +255,12 @@ defined probabilities $$ \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), \end{align*} $$ -where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). +where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).











@@ -268,22 +268,22 @@ where \( \hat{\beta} \) are the weights we wish to extract from data, in our cas

The equations to solve

-Our compact equations used a definition of a vector \( \hat{y} \) with \( n \) -elements \( y_i \), an \( n\times p \) matrix \( \hat{X} \) which contains the -\( x_i \) values and a vector \( \hat{p} \) of fitted probabilities -\( p(y_i\vert x_i,\hat{\beta}) \). We rewrote in a more compact form +Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) +elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the +\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities +\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form the first derivative of the cost function as $$ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$

-If we in addition define a diagonal matrix \( \hat{W} \) with elements -\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as $$ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$ This defines what is called the Hessian matrix. @@ -300,13 +300,13 @@ If we can set up these equations, Newton-Raphson's iterative method is normally Our iterative scheme is then given by $$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T}\right)^{-1}_{\hat{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}}\right)_{\hat{\beta}^{\mathrm{old}}}, +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, $$ or in matrix form as $$ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\hat{X}^T\hat{W}\hat{X} \right)^{-1}\times \left(-\hat{X}^T(\hat{y}-\hat{p}) \right)_{\hat{\beta}^{\mathrm{old}}}. +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. $$ The right-hand side is computed with the old values of \( \beta \). @@ -410,9 +410,9 @@ $$ \end{array}. $$ -Defining the Jacobian matrix \( {\bf \hat{J}} \) we have +Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have $$ - {\bf \hat{J}}=\left( \begin{array}{cc} + {\bf \boldsymbol{J}}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ \partial f_2/\partial x_1 &\partial f_2/\partial x_2 \end{array} \right), @@ -428,13 +428,13 @@ $$ where we have defined $$ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= - -{\bf \hat{J}}^{-1} + -{\bf \boldsymbol{J}}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). $$ We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may -arise in case \( {\bf \hat{J}} \) is nearly singular. +arise in case \( {\bf \boldsymbol{J}} \) is nearly singular.

It is rather straightforward to extend the above scheme to systems of @@ -662,7 +662,7 @@ Using the definition of convexity, try to show that a function satisfying the pr

Before we proceed, we would like to discuss the approach called the -standard Steepest descent, which again leads to us having to be able +standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).

@@ -673,7 +673,7 @@ the class of iterative methods for solving problems from linear algebra of the type $$ \begin{equation*} -\hat{A}\hat{x} = \hat{b}. +\boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}. \end{equation*} $$ @@ -682,14 +682,14 @@ In the iterative process we end up with a problem like $$ \begin{equation*} - \hat{r}= \hat{b}-\hat{A}\hat{x}, + \boldsymbol{r}= \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}, \end{equation*} $$ -where \( \hat{r} \) is the so-called residual or error in the iterative process. +where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.

-When we have found the exact solution, \( \hat{r}=0 \). +When we have found the exact solution, \( \boldsymbol{r}=0 \).











@@ -700,12 +700,12 @@ When we have found the exact solution, \( \hat{r}=0 \). The residual is zero when we reach the minimum of the quadratic equation $$ \begin{equation*} - P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b}, + P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b}, \end{equation*} $$

-with the constraint that the matrix \( \hat{A} \) is positive definite and +with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and symmetric. This defines also the Hessian and we want it to be positive definite.

@@ -714,18 +714,18 @@ symmetric. This defines also the Hessian and we want it to be positive definit

Steepest descent method

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that $$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$ or consider the system $$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ @@ -738,24 +738,24 @@ instead.

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form $$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \hat{r}_1 \) (see below for definition) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals $$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \).

@@ -771,34 +771,34 @@ and We can compute the residual iteratively as $$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ which equals $$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{r}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k), \end{equation*} $$ or $$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{r}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k, \end{equation*} $$ which gives $$ -\alpha_k = \frac{\hat{r}_k^T\hat{r}_k}{\hat{r}_k^T\hat{A}\hat{r}_k} +\alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k} $$ leading to the iterative scheme $$ \begin{equation*} -\hat{x}_{k+1}=\hat{x}_k-\alpha_k\hat{r}_{k}, +\boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k}, \end{equation*} $$ @@ -881,25 +881,25 @@ pt.plot(it_array

In the CG method we define so-called conjugate directions and two vectors -\( \hat{s} \) and \( \hat{t} \) +\( \boldsymbol{s} \) and \( \boldsymbol{t} \) are said to be conjugate if $$ \begin{equation*} -\hat{s}^T\hat{A}\hat{t}= 0. +\boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0. \end{equation*} $$ The philosophy of the CG method is to perform searches in various conjugate directions -of our vectors \( \hat{x}_i \) obeying the above criterion, namely +of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely $$ \begin{equation*} -\hat{x}_i^T\hat{A}\hat{x}_j= 0. +\boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0. \end{equation*} $$ Two vectors are conjugate if they are orthogonal with respect to -this inner product. Being conjugate is a symmetric relation: if \( \hat{s} \) is conjugate to \( \hat{t} \), then \( \hat{t} \) is conjugate to \( \hat{s} \). +this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \). @@ -913,7 +913,7 @@ this inner product. Being conjugate is a symmetric relation: if \( \hat{s} \) is An example is given by the eigenvectors of the matrix $$ \begin{equation*} -\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j, +\boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j, \end{equation*} $$ @@ -928,21 +928,21 @@ which is zero unless \( i=j \).

-Assume now that we have a symmetric positive-definite matrix \( \hat{A} \) of size +Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size \( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector $$ \begin{equation*} -\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}. +\boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}. \end{equation*} $$ -We assume that \( \hat{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. -Then the \( \hat{p}_{i} \) form a basis of \( R^n \) and we can expand the solution -$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely +We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. +Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution +$ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely $$ \begin{equation*} - \hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i. + \boldsymbol{x} = \sum^{n}_{i=1} \alpha_i \boldsymbol{p}_i. \end{equation*} $$

@@ -962,11 +962,11 @@ $$ \end{equation*} $$ -Multiplying with \( \hat{p}_k^T \) from the left gives +Multiplying with \( \boldsymbol{p}_k^T \) from the left gives $$ \begin{equation*} - \hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b}, + \boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{x} = \sum^{n}_{i=1} \alpha_i\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{p}_i= \boldsymbol{p}_k^T \boldsymbol{b}, \end{equation*} $$ @@ -974,7 +974,7 @@ and we can define the coefficients \( \alpha_k \) as $$ \begin{equation*} - \alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k} + \alpha_k = \frac{\boldsymbol{p}_k^T \boldsymbol{b}}{\boldsymbol{p}_k^T \boldsymbol{A} \boldsymbol{p}_k} \end{equation*} $$ @@ -989,26 +989,26 @@ $$

-If we choose the conjugate vectors \( \hat{p}_k \) carefully, +If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, then we may not need all of them to obtain a good approximation to the solution -\( \hat{x} \). +\( \boldsymbol{x} \). We want to regard the conjugate gradient method as an iterative method. This will us to solve systems where \( n \) is so large that the direct method would take too much time.

-We denote the initial guess for \( \hat{x} \) as \( \hat{x}_0 \). +We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that $$ \begin{equation*} -\hat{x}_0=0, +\boldsymbol{x}_0=0, \end{equation*} $$ or consider the system $$ \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ @@ -1023,24 +1023,24 @@ instead.

-One can show that the solution \( \hat{x} \) is also the unique minimizer of the quadratic form +One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form $$ \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \hat{p}_1 \) -to be the gradient of \( f \) at \( \hat{x}=\hat{x}_0 \), +This suggests taking the first basis vector \( \boldsymbol{p}_1 \) +to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals $$ \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ and -\( \hat{x}_0=0 \) it is equal \( -\hat{b} \). +\( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method.

@@ -1053,23 +1053,23 @@ hence the name conjugate gradient method.

-Let \( \hat{r}_k \) be the residual at the \( k \)-th step: +Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step: $$ \begin{equation*} -\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k. +\boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k. \end{equation*} $$ -Note that \( \hat{r}_k \) is the negative gradient of \( f \) at -\( \hat{x}=\hat{x}_k \), -so the gradient descent method would be to move in the direction \( \hat{r}_k \). -Here, we insist that the directions \( \hat{p}_k \) are conjugate to each other, -so we take the direction closest to the gradient \( \hat{r}_k \) +Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at +\( \boldsymbol{x}=\boldsymbol{x}_k \), +so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \). +Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other, +so we take the direction closest to the gradient \( \boldsymbol{r}_k \) under the conjugacy constraint. This gives the following expression $$ \begin{equation*} -\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k. +\boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k. \end{equation*} $$

@@ -1085,21 +1085,21 @@ $$ We can also compute the residual iteratively as $$ \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ which equals $$ \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k), +\boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k), \end{equation*} $$ or $$ \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k, +(\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k, \end{equation*} $$ @@ -1107,7 +1107,7 @@ which gives $$ \begin{equation*} -\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k}, +\boldsymbol{r}_{k+1}=\boldsymbol{r}_k-\boldsymbol{A}\boldsymbol{p}_{k}, \end{equation*} $$ @@ -1138,12 +1138,12 @@ $$ with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by $$ -h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, +h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, $$ such that $$ -\hat{y}_i = \beta_0 + \beta_1 x_i. +\boldsymbol{y}_i = \beta_0 + \beta_1 x_i. $$

@@ -1152,10 +1152,10 @@ $$

Gradient descent example

-Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) +Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

-It is convenient to write \( \mathbf{\hat{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by $$ X \equiv \begin{bmatrix} 1 &; x_1 \\ @@ -1192,7 +1192,7 @@ where \( X \) is the design matrix defined above.

The Hessian matrix

The Hessian matrix of \( C(\beta) \) is given by $$ -\hat{H} \equiv \begin{bmatrix} +\boldsymbol{H} \equiv \begin{bmatrix} \frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ \end{bmatrix} = 2X^T X. @@ -1849,9 +1849,9 @@ $$ \label{_auto4}\\ \mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ \mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ -\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \label{_auto5} \end{align} $$ @@ -1865,12 +1865,12 @@ respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance -\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - -(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - +(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by $$ -\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. +\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$

diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index 161bfdccd800e346b0e3be453ac724a31cf1f287..09b10088679ff91d763e7c7a6b4846f61988dbf8 100644 GIT binary patch literal 192 zcmV;x06+g9iwFRzVQpUk1MSaC3c@fD2H>uHia9|^nx@62U>7a~5igL^sf~}MCMnw6 z+Xv`MaZ^OdxA_@n7-n|)a<$1LcXz>T2q9^bz>rzW$0YGok0|9pS;knLl9W>*lrf$J zK;}E?rL|6&{*?NN%A$HZ*N;`zhdt9P@XSB4ucd|cwsW;sptQq6>l54%Pizs%rc(u! u8tu>mgV$CX1fe<*WhtH1N}R*S=z|d#8->4q#`8SS``QCR?OMA42mk=U9$Ucx literal 191 zcmV;w06_mAiwFP+U~FFi1MSbv3c@f92k@Qu6nTP?*?r(qa0d^9h%eBw%++Z-+Ya5^ zyARNl;$?`?-{nt82+2NMt+!cV@79|QA|_D;L#9bQCN8FW6oxT`EM*LnzCgf`F_wCO z^PTk48q0NmN_8S>NWPux$I9} \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 24, 2020**\n", + "Date: **Sep 25, 2020**\n", "\n", "Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -56,8 +56,8 @@ "source": [ "$$\n", "\\begin{align*}\n", - "p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n", - "p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n", + "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n", + "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n", "\\end{align*}\n", "$$" ] @@ -66,14 +66,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", + "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", "\n", "## The equations to solve\n", "\n", - "Our compact equations used a definition of a vector $\\hat{y}$ with $n$\n", - "elements $y_i$, an $n\\times p$ matrix $\\hat{X}$ which contains the\n", - "$x_i$ values and a vector $\\hat{p}$ of fitted probabilities\n", - "$p(y_i\\vert x_i,\\hat{\\beta})$. We rewrote in a more compact form\n", + "Our compact equations used a definition of a vector $\\boldsymbol{y}$ with $n$\n", + "elements $y_i$, an $n\\times p$ matrix $\\boldsymbol{X}$ which contains the\n", + "$x_i$ values and a vector $\\boldsymbol{p}$ of fitted probabilities\n", + "$p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We rewrote in a more compact form\n", "the first derivative of the cost function as" ] }, @@ -82,7 +82,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", "$$" ] }, @@ -90,8 +90,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If we in addition define a diagonal matrix $\\hat{W}$ with elements \n", - "$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as" + "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", + "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" ] }, { @@ -99,7 +99,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n", + "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", "$$" ] }, @@ -121,7 +121,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\beta}^{\\mathrm{new}} = \\hat{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T}\\right)^{-1}_{\\hat{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}}\\right)_{\\hat{\\beta}^{\\mathrm{old}}},\n", + "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", "$$" ] }, @@ -137,7 +137,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\beta}^{\\mathrm{new}} = \\hat{\\beta}^{\\mathrm{old}}-\\left(\\hat{X}^T\\hat{W}\\hat{X} \\right)^{-1}\\times \\left(-\\hat{X}^T(\\hat{y}-\\hat{p}) \\right)_{\\hat{\\beta}^{\\mathrm{old}}}.\n", + "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", "$$" ] }, @@ -294,7 +294,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Defining the Jacobian matrix ${\\bf \\hat{J}}$ we have" + "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { @@ -302,7 +302,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\bf \\hat{J}}=\\left( \\begin{array}{cc}\n", + "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", " \\partial f_1/\\partial x_1 & \\partial f_1/\\partial x_2 \\\\\n", " \\partial f_2/\\partial x_1 &\\partial f_2/\\partial x_2\n", " \\end{array} \\right),\n", @@ -340,7 +340,7 @@ "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", - " -{\\bf \\hat{J}}^{-1}\n", + " -{\\bf \\boldsymbol{J}}^{-1}\n", " \\left(\\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\\\ f_2(x_1^{n},x_2^{n}) \\end{array} \\right).\n", "$$" ] @@ -351,7 +351,7 @@ "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", - "arise in case ${\\bf \\hat{J}}$ is nearly singular.\n", + "arise in case ${\\bf \\boldsymbol{J}}$ is nearly singular.\n", "\n", "It is rather straightforward to extend the above scheme to systems of\n", "more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. \n", @@ -552,7 +552,7 @@ "\n", "\n", "Before we proceed, we would like to discuss the approach called the\n", - "**standard Steepest descent**, which again leads to us having to be able\n", + "**standard Steepest descent** (different from the above steepest descent discussion), which again leads to us having to be able\n", "to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).\n", "\n", "[The success of the CG method](https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf)\n", @@ -567,7 +567,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{A}\\hat{x} = \\hat{b}.\n", + "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", "$$" ] }, @@ -583,7 +583,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{r}= \\hat{b}-\\hat{A}\\hat{x},\n", + "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", "$$" ] }, @@ -591,9 +591,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\hat{r}$ is the so-called residual or error in the iterative process.\n", + "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", - "When we have found the exact solution, $\\hat{r}=0$.\n", + "When we have found the exact solution, $\\boldsymbol{r}=0$.\n", "\n", "## Gradient method\n", "\n", @@ -605,7 +605,7 @@ "metadata": {}, "source": [ "$$\n", - "P(\\hat{x})=\\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T\\hat{b},\n", + "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", "$$" ] }, @@ -613,13 +613,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "with the constraint that the matrix $\\hat{A}$ is positive definite and\n", + "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite. \n", "\n", "\n", "## Steepest descent method\n", "\n", - "We denote the initial guess for $\\hat{x}$ as $\\hat{x}_0$. \n", + "We denote the initial guess for $\\boldsymbol{x}$ as $\\boldsymbol{x}_0$. \n", "We can assume without loss of generality that" ] }, @@ -628,7 +628,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{x}_0=0,\n", + "\\boldsymbol{x}_0=0,\n", "$$" ] }, @@ -644,7 +644,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{A}\\hat{z} = \\hat{b}-\\hat{A}\\hat{x}_0,\n", + "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", "$$" ] }, @@ -656,7 +656,7 @@ "\n", "\n", "## Steepest descent method\n", - "One can show that the solution $\\hat{x}$ is also the unique minimizer of the quadratic form" + "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" ] }, { @@ -664,7 +664,7 @@ "metadata": {}, "source": [ "$$\n", - "f(\\hat{x}) = \\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T \\hat{x} , \\quad \\hat{x}\\in\\mathbf{R}^n.\n", + "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", "$$" ] }, @@ -672,8 +672,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This suggests taking the first basis vector $\\hat{r}_1$ (see below for definition) \n", - "to be the gradient of $f$ at $\\hat{x}=\\hat{x}_0$, \n", + "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", + "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", "which equals" ] }, @@ -682,7 +682,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{A}\\hat{x}_0-\\hat{b},\n", + "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", "$$" ] }, @@ -691,7 +691,7 @@ "metadata": {}, "source": [ "and \n", - "$\\hat{x}_0=0$ it is equal $-\\hat{b}$.\n", + "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", "\n", "\n", "\n", @@ -704,7 +704,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{r}_{k+1}=\\hat{b}-\\hat{A}\\hat{x}_{k+1},\n", + "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", "$$" ] }, @@ -720,7 +720,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{b}-\\hat{A}(\\hat{x}_k+\\alpha_k\\hat{r}_k),\n", + "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", "$$" ] }, @@ -736,7 +736,7 @@ "metadata": {}, "source": [ "$$\n", - "(\\hat{b}-\\hat{A}\\hat{x}_k)-\\alpha_k\\hat{A}\\hat{r}_k,\n", + "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", "$$" ] }, @@ -752,7 +752,7 @@ "metadata": {}, "source": [ "$$\n", - "\\alpha_k = \\frac{\\hat{r}_k^T\\hat{r}_k}{\\hat{r}_k^T\\hat{A}\\hat{r}_k}\n", + "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", "$$" ] }, @@ -768,7 +768,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{x}_{k+1}=\\hat{x}_k-\\alpha_k\\hat{r}_{k},\n", + "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k-\\alpha_k\\boldsymbol{r}_{k},\n", "$$" ] }, @@ -901,7 +901,7 @@ "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", - "$\\hat{s}$ and $\\hat{t}$\n", + "$\\boldsymbol{s}$ and $\\boldsymbol{t}$\n", "are said to be\n", "conjugate if" ] @@ -911,7 +911,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{s}^T\\hat{A}\\hat{t}= 0.\n", + "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", "$$" ] }, @@ -920,7 +920,7 @@ "metadata": {}, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", - "of our vectors $\\hat{x}_i$ obeying the above criterion, namely" + "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" ] }, { @@ -928,7 +928,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{x}_i^T\\hat{A}\\hat{x}_j= 0.\n", + "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", "$$" ] }, @@ -937,7 +937,7 @@ "metadata": {}, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", - "this inner product. Being conjugate is a symmetric relation: if $\\hat{s}$ is conjugate to $\\hat{t}$, then $\\hat{t}$ is conjugate to $\\hat{s}$.\n", + "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$.\n", "\n", "\n", "\n", @@ -950,7 +950,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{v}_i^T\\hat{A}\\hat{v}_j= \\lambda\\hat{v}_i^T\\hat{v}_j,\n", + "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", "$$" ] }, @@ -964,7 +964,7 @@ "\n", "\n", "## Conjugate gradient method\n", - "Assume now that we have a symmetric positive-definite matrix $\\hat{A}$ of size\n", + "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", "$n\\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector" ] }, @@ -973,7 +973,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{x}_{i+1}=\\hat{x}_{i}+\\alpha_i\\hat{p}_{i}.\n", + "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", "$$" ] }, @@ -981,9 +981,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We assume that $\\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", - "Then the $\\hat{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", - "$ \\hat{A}\\hat{x} = \\hat{b}$ in this basis, namely" + "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", + "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", + "$ \\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}$ in this basis, namely" ] }, { @@ -991,7 +991,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{x} = \\sum^{n}_{i=1} \\alpha_i \\hat{p}_i.\n", + "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", "$$" ] }, @@ -1016,7 +1016,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Multiplying with $\\hat{p}_k^T$ from the left gives" + "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] }, { @@ -1024,7 +1024,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{p}_k^T \\hat{A}\\hat{x} = \\sum^{n}_{i=1} \\alpha_i\\hat{p}_k^T \\hat{A}\\hat{p}_i= \\hat{p}_k^T \\hat{b},\n", + "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", "$$" ] }, @@ -1040,7 +1040,7 @@ "metadata": {}, "source": [ "$$\n", - "\\alpha_k = \\frac{\\hat{p}_k^T \\hat{b}}{\\hat{p}_k^T \\hat{A} \\hat{p}_k}\n", + "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", "$$" ] }, @@ -1050,14 +1050,14 @@ "source": [ "## Conjugate gradient method and iterations\n", "\n", - "If we choose the conjugate vectors $\\hat{p}_k$ carefully, \n", + "If we choose the conjugate vectors $\\boldsymbol{p}_k$ carefully, \n", "then we may not need all of them to obtain a good approximation to the solution \n", - "$\\hat{x}$. \n", + "$\\boldsymbol{x}$. \n", "We want to regard the conjugate gradient method as an iterative method. \n", "This will us to solve systems where $n$ is so large that the direct \n", "method would take too much time.\n", "\n", - "We denote the initial guess for $\\hat{x}$ as $\\hat{x}_0$. \n", + "We denote the initial guess for $\\boldsymbol{x}$ as $\\boldsymbol{x}_0$. \n", "We can assume without loss of generality that" ] }, @@ -1066,7 +1066,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{x}_0=0,\n", + "\\boldsymbol{x}_0=0,\n", "$$" ] }, @@ -1082,7 +1082,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{A}\\hat{z} = \\hat{b}-\\hat{A}\\hat{x}_0,\n", + "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", "$$" ] }, @@ -1096,7 +1096,7 @@ "\n", "\n", "## Conjugate gradient method\n", - "One can show that the solution $\\hat{x}$ is also the unique minimizer of the quadratic form" + "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" ] }, { @@ -1104,7 +1104,7 @@ "metadata": {}, "source": [ "$$\n", - "f(\\hat{x}) = \\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T \\hat{x} , \\quad \\hat{x}\\in\\mathbf{R}^n.\n", + "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", "$$" ] }, @@ -1112,8 +1112,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This suggests taking the first basis vector $\\hat{p}_1$ \n", - "to be the gradient of $f$ at $\\hat{x}=\\hat{x}_0$, \n", + "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", + "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", "which equals" ] }, @@ -1122,7 +1122,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{A}\\hat{x}_0-\\hat{b},\n", + "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", "$$" ] }, @@ -1131,7 +1131,7 @@ "metadata": {}, "source": [ "and \n", - "$\\hat{x}_0=0$ it is equal $-\\hat{b}$.\n", + "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", "The other vectors in the basis will be conjugate to the gradient, \n", "hence the name conjugate gradient method.\n", "\n", @@ -1139,7 +1139,7 @@ "\n", "\n", "## Conjugate gradient method\n", - "Let $\\hat{r}_k$ be the residual at the $k$-th step:" + "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" ] }, { @@ -1147,7 +1147,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{r}_k=\\hat{b}-\\hat{A}\\hat{x}_k.\n", + "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", "$$" ] }, @@ -1155,11 +1155,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Note that $\\hat{r}_k$ is the negative gradient of $f$ at \n", - "$\\hat{x}=\\hat{x}_k$, \n", - "so the gradient descent method would be to move in the direction $\\hat{r}_k$. \n", - "Here, we insist that the directions $\\hat{p}_k$ are conjugate to each other, \n", - "so we take the direction closest to the gradient $\\hat{r}_k$ \n", + "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", + "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", + "so the gradient descent method would be to move in the direction $\\boldsymbol{r}_k$. \n", + "Here, we insist that the directions $\\boldsymbol{p}_k$ are conjugate to each other, \n", + "so we take the direction closest to the gradient $\\boldsymbol{r}_k$ \n", "under the conjugacy constraint. \n", "This gives the following expression" ] @@ -1169,7 +1169,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{p}_{k+1}=\\hat{r}_k-\\frac{\\hat{p}_k^T \\hat{A}\\hat{r}_k}{\\hat{p}_k^T\\hat{A}\\hat{p}_k} \\hat{p}_k.\n", + "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", "$$" ] }, @@ -1186,7 +1186,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{r}_{k+1}=\\hat{b}-\\hat{A}\\hat{x}_{k+1},\n", + "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", "$$" ] }, @@ -1202,7 +1202,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{b}-\\hat{A}(\\hat{x}_k+\\alpha_k\\hat{p}_k),\n", + "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", "$$" ] }, @@ -1218,7 +1218,7 @@ "metadata": {}, "source": [ "$$\n", - "(\\hat{b}-\\hat{A}\\hat{x}_k)-\\alpha_k\\hat{A}\\hat{p}_k,\n", + "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", "$$" ] }, @@ -1234,7 +1234,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{r}_{k+1}=\\hat{r}_k-\\hat{A}\\hat{p}_{k},\n", + "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", "$$" ] }, @@ -1281,7 +1281,7 @@ "metadata": {}, "source": [ "$$\n", - "h_\\beta(x) = \\hat{y} = \\beta_0 + \\beta_1 x,\n", + "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", "$$" ] }, @@ -1297,7 +1297,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{y}_i = \\beta_0 + \\beta_1 x_i.\n", + "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", "$$" ] }, @@ -1308,9 +1308,9 @@ "\n", "## Gradient descent example\n", "\n", - "Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\hat{y}} = (\\hat{y}_1,\\cdots,\\hat{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n", + "Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\boldsymbol{y}} = (\\boldsymbol{y}_1,\\cdots,\\boldsymbol{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n", "\n", - "It is convenient to write $\\mathbf{\\hat{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by" + "It is convenient to write $\\mathbf{\\boldsymbol{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by" ] }, { @@ -1379,7 +1379,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{H} \\equiv \\begin{bmatrix}\n", + "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0^2} &; \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} \\\\\n", "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} &; \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_1^2} &; \\\\\n", "\\end{bmatrix} = 2X^T X.\n", @@ -2266,7 +2266,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", + "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", "$$" ] }, @@ -2275,7 +2275,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", + "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", "$$" ] }, @@ -2284,7 +2284,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\hat{\\mathbf{m}}_t \\over \\sqrt{\\hat{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", + "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", "$$" ] }, @@ -2313,8 +2313,8 @@ "Like in RMSprop, the effective step size of a parameter depends on the\n", "magnitude of its gradient squared. To understand this better, let us\n", "rewrite this expression in terms of the variance\n", - "$\\boldsymbol{\\sigma}_t^2 = \\hat{\\mathbf{s}}_t -\n", - "(\\hat{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n", + "$\\boldsymbol{\\sigma}_t^2 = \\boldsymbol{\\mathbf{s}}_t -\n", + "(\\boldsymbol{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n", "update rule for this parameter is given by" ] }, @@ -2323,7 +2323,7 @@ "metadata": {}, "source": [ "$$\n", - "\\Delta \\theta_{t+1}= -\\eta_t { \\hat{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", + "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", "$$" ] }, diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index 75535d279..d39a61535 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -41,33 +41,33 @@ defined probabilities !bt \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}), \end{align*} !et -where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. +where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. !split ===== The equations to solve ===== -Our compact equations used a definition of a vector $\hat{y}$ with $n$ -elements $y_i$, an $n\times p$ matrix $\hat{X}$ which contains the -$x_i$ values and a vector $\hat{p}$ of fitted probabilities -$p(y_i\vert x_i,\hat{\beta})$. We rewrote in a more compact form +Our compact equations used a definition of a vector $\bm{y}$ with $n$ +elements $y_i$, an $n\times p$ matrix $\bm{X}$ which contains the +$x_i$ values and a vector $\bm{p}$ of fitted probabilities +$p(y_i\vert x_i,\bm{\beta})$. We rewrote in a more compact form the first derivative of the cost function as !bt \[ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right). \] !et -If we in addition define a diagonal matrix $\hat{W}$ with elements -$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix $\bm{W}$ with elements +$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as !bt \[ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}. \] !et This defines what is called the Hessian matrix. @@ -81,14 +81,14 @@ Our iterative scheme is then given by !bt \[ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T}\right)^{-1}_{\hat{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}}\right)_{\hat{\beta}^{\mathrm{old}}}, +\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T}\right)^{-1}_{\bm{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}}\right)_{\bm{\beta}^{\mathrm{old}}}, \] !et or in matrix form as !bt \[ -\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\hat{X}^T\hat{W}\hat{X} \right)^{-1}\times \left(-\hat{X}^T(\hat{y}-\hat{p}) \right)_{\hat{\beta}^{\mathrm{old}}}. +\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\bm{X}^T\bm{W}\bm{X} \right)^{-1}\times \left(-\bm{X}^T(\bm{y}-\bm{p}) \right)_{\bm{\beta}^{\mathrm{old}}}. \] !et The right-hand side is computed with the old values of $\beta$. @@ -189,10 +189,10 @@ which we Taylor expand to obtain \end{array}. \] !et -Defining the Jacobian matrix ${\bf \hat{J}}$ we have +Defining the Jacobian matrix ${\bf \bm{J}}$ we have !bt \[ - {\bf \hat{J}}=\left( \begin{array}{cc} + {\bf \bm{J}}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ \partial f_2/\partial x_1 &\partial f_2/\partial x_2 \end{array} \right), @@ -210,13 +210,13 @@ where we have defined !bt \[ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= - -{\bf \hat{J}}^{-1} + -{\bf \bm{J}}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). \] !et We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may -arise in case ${\bf \hat{J}}$ is nearly singular. +arise in case ${\bf \bm{J}}$ is nearly singular. It is rather straightforward to extend the above scheme to systems of more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. @@ -391,7 +391,7 @@ Using the definition of convexity, try to show that a function satisfying the pr Before we proceed, we would like to discuss the approach called the -_standard Steepest descent_, which again leads to us having to be able +_standard Steepest descent_ (different from the above steepest descent discussion), which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG). "The success of the CG method":"https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf" @@ -401,7 +401,7 @@ the class of iterative methods for solving problems from linear algebra of the type !bt \begin{equation*} -\hat{A}\hat{x} = \hat{b}. +\bm{A}\bm{x} = \bm{b}. \end{equation*} !et @@ -409,12 +409,12 @@ In the iterative process we end up with a problem like !bt \begin{equation*} - \hat{r}= \hat{b}-\hat{A}\hat{x}, + \bm{r}= \bm{b}-\bm{A}\bm{x}, \end{equation*} !et -where $\hat{r}$ is the so-called residual or error in the iterative process. +where $\bm{r}$ is the so-called residual or error in the iterative process. -When we have found the exact solution, $\hat{r}=0$. +When we have found the exact solution, $\bm{r}=0$. !split ===== Gradient method ===== @@ -422,28 +422,28 @@ When we have found the exact solution, $\hat{r}=0$. The residual is zero when we reach the minimum of the quadratic equation !bt \begin{equation*} - P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b}, + P(\bm{x})=\frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T\bm{b}, \end{equation*} !et -with the constraint that the matrix $\hat{A}$ is positive definite and +with the constraint that the matrix $\bm{A}$ is positive definite and symmetric. This defines also the Hessian and we want it to be positive definite. !split ===== Steepest descent method ===== -We denote the initial guess for $\hat{x}$ as $\hat{x}_0$. +We denote the initial guess for $\bm{x}$ as $\bm{x}_0$. We can assume without loss of generality that !bt \begin{equation*} -\hat{x}_0=0, +\bm{x}_0=0, \end{equation*} !et or consider the system !bt \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\bm{A}\bm{z} = \bm{b}-\bm{A}\bm{x}_0, \end{equation*} !et instead. @@ -452,22 +452,22 @@ instead. !split ===== Steepest descent method ===== !bblock -One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form +One can show that the solution $\bm{x}$ is also the unique minimizer of the quadratic form !bt \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\bm{x}) = \frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T \bm{x} , \quad \bm{x}\in\mathbf{R}^n. \end{equation*} !et -This suggests taking the first basis vector $\hat{r}_1$ (see below for definition) -to be the gradient of $f$ at $\hat{x}=\hat{x}_0$, +This suggests taking the first basis vector $\bm{r}_1$ (see below for definition) +to be the gradient of $f$ at $\bm{x}=\bm{x}_0$, which equals !bt \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\bm{A}\bm{x}_0-\bm{b}, \end{equation*} !et and -$\hat{x}_0=0$ it is equal $-\hat{b}$. +$\bm{x}_0=0$ it is equal $-\bm{b}$. !eblock @@ -477,32 +477,32 @@ $\hat{x}_0=0$ it is equal $-\hat{b}$. We can compute the residual iteratively as !bt \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\bm{r}_{k+1}=\bm{b}-\bm{A}\bm{x}_{k+1}, \end{equation*} !et which equals !bt \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{r}_k), +\bm{b}-\bm{A}(\bm{x}_k+\alpha_k\bm{r}_k), \end{equation*} !et or !bt \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{r}_k, +(\bm{b}-\bm{A}\bm{x}_k)-\alpha_k\bm{A}\bm{r}_k, \end{equation*} !et which gives !bt \[ -\alpha_k = \frac{\hat{r}_k^T\hat{r}_k}{\hat{r}_k^T\hat{A}\hat{r}_k} +\alpha_k = \frac{\bm{r}_k^T\bm{r}_k}{\bm{r}_k^T\bm{A}\bm{r}_k} \] !et leading to the iterative scheme !bt \begin{equation*} -\hat{x}_{k+1}=\hat{x}_k-\alpha_k\hat{r}_{k}, +\bm{x}_{k+1}=\bm{x}_k-\alpha_k\bm{r}_{k}, \end{equation*} !et !eblock @@ -567,23 +567,23 @@ pt.plot(it_array.T[0], it_array.T[1], "x-") ===== Conjugate gradient method ===== !bblock In the CG method we define so-called conjugate directions and two vectors -$\hat{s}$ and $\hat{t}$ +$\bm{s}$ and $\bm{t}$ are said to be conjugate if !bt \begin{equation*} -\hat{s}^T\hat{A}\hat{t}= 0. +\bm{s}^T\bm{A}\bm{t}= 0. \end{equation*} !et The philosophy of the CG method is to perform searches in various conjugate directions -of our vectors $\hat{x}_i$ obeying the above criterion, namely +of our vectors $\bm{x}_i$ obeying the above criterion, namely !bt \begin{equation*} -\hat{x}_i^T\hat{A}\hat{x}_j= 0. +\bm{x}_i^T\bm{A}\bm{x}_j= 0. \end{equation*} !et Two vectors are conjugate if they are orthogonal with respect to -this inner product. Being conjugate is a symmetric relation: if $\hat{s}$ is conjugate to $\hat{t}$, then $\hat{t}$ is conjugate to $\hat{s}$. +this inner product. Being conjugate is a symmetric relation: if $\bm{s}$ is conjugate to $\bm{t}$, then $\bm{t}$ is conjugate to $\bm{s}$. !eblock !split @@ -592,7 +592,7 @@ this inner product. Being conjugate is a symmetric relation: if $\hat{s}$ is con An example is given by the eigenvectors of the matrix !bt \begin{equation*} -\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j, +\bm{v}_i^T\bm{A}\bm{v}_j= \lambda\bm{v}_i^T\bm{v}_j, \end{equation*} !et which is zero unless $i=j$. @@ -602,20 +602,20 @@ which is zero unless $i=j$. !split ===== Conjugate gradient method ===== !bblock -Assume now that we have a symmetric positive-definite matrix $\hat{A}$ of size +Assume now that we have a symmetric positive-definite matrix $\bm{A}$ of size $n\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector !bt \begin{equation*} -\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}. +\bm{x}_{i+1}=\bm{x}_{i}+\alpha_i\bm{p}_{i}. \end{equation*} !et -We assume that $\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions. -Then the $\hat{p}_{i}$ form a basis of $R^n$ and we can expand the solution -$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely +We assume that $\bm{p}_{i}$ is a sequence of $n$ mutually conjugate directions. +Then the $\bm{p}_{i}$ form a basis of $R^n$ and we can expand the solution +$ \bm{A}\bm{x} = \bm{b}$ in this basis, namely !bt \begin{equation*} - \hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i. + \bm{x} = \sum^{n}_{i=1} \alpha_i \bm{p}_i. \end{equation*} !et !eblock @@ -629,18 +629,18 @@ The coefficients are given by \mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}. \end{equation*} !et -Multiplying with $\hat{p}_k^T$ from the left gives +Multiplying with $\bm{p}_k^T$ from the left gives !bt \begin{equation*} - \hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b}, + \bm{p}_k^T \bm{A}\bm{x} = \sum^{n}_{i=1} \alpha_i\bm{p}_k^T \bm{A}\bm{p}_i= \bm{p}_k^T \bm{b}, \end{equation*} !et and we can define the coefficients $\alpha_k$ as !bt \begin{equation*} - \alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k} + \alpha_k = \frac{\bm{p}_k^T \bm{b}}{\bm{p}_k^T \bm{A} \bm{p}_k} \end{equation*} !et !eblock @@ -649,24 +649,24 @@ and we can define the coefficients $\alpha_k$ as ===== Conjugate gradient method and iterations ===== !bblock -If we choose the conjugate vectors $\hat{p}_k$ carefully, +If we choose the conjugate vectors $\bm{p}_k$ carefully, then we may not need all of them to obtain a good approximation to the solution -$\hat{x}$. +$\bm{x}$. We want to regard the conjugate gradient method as an iterative method. This will us to solve systems where $n$ is so large that the direct method would take too much time. -We denote the initial guess for $\hat{x}$ as $\hat{x}_0$. +We denote the initial guess for $\bm{x}$ as $\bm{x}_0$. We can assume without loss of generality that !bt \begin{equation*} -\hat{x}_0=0, +\bm{x}_0=0, \end{equation*} !et or consider the system !bt \begin{equation*} -\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\bm{A}\bm{z} = \bm{b}-\bm{A}\bm{x}_0, \end{equation*} !et instead. @@ -676,22 +676,22 @@ instead. !split ===== Conjugate gradient method ===== !bblock -One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form +One can show that the solution $\bm{x}$ is also the unique minimizer of the quadratic form !bt \begin{equation*} - f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. + f(\bm{x}) = \frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T \bm{x} , \quad \bm{x}\in\mathbf{R}^n. \end{equation*} !et -This suggests taking the first basis vector $\hat{p}_1$ -to be the gradient of $f$ at $\hat{x}=\hat{x}_0$, +This suggests taking the first basis vector $\bm{p}_1$ +to be the gradient of $f$ at $\bm{x}=\bm{x}_0$, which equals !bt \begin{equation*} -\hat{A}\hat{x}_0-\hat{b}, +\bm{A}\bm{x}_0-\bm{b}, \end{equation*} !et and -$\hat{x}_0=0$ it is equal $-\hat{b}$. +$\bm{x}_0=0$ it is equal $-\bm{b}$. The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method. !eblock @@ -700,22 +700,22 @@ hence the name conjugate gradient method. !split ===== Conjugate gradient method ===== !bblock -Let $\hat{r}_k$ be the residual at the $k$-th step: +Let $\bm{r}_k$ be the residual at the $k$-th step: !bt \begin{equation*} -\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k. +\bm{r}_k=\bm{b}-\bm{A}\bm{x}_k. \end{equation*} !et -Note that $\hat{r}_k$ is the negative gradient of $f$ at -$\hat{x}=\hat{x}_k$, -so the gradient descent method would be to move in the direction $\hat{r}_k$. -Here, we insist that the directions $\hat{p}_k$ are conjugate to each other, -so we take the direction closest to the gradient $\hat{r}_k$ +Note that $\bm{r}_k$ is the negative gradient of $f$ at +$\bm{x}=\bm{x}_k$, +so the gradient descent method would be to move in the direction $\bm{r}_k$. +Here, we insist that the directions $\bm{p}_k$ are conjugate to each other, +so we take the direction closest to the gradient $\bm{r}_k$ under the conjugacy constraint. This gives the following expression !bt \begin{equation*} -\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k. +\bm{p}_{k+1}=\bm{r}_k-\frac{\bm{p}_k^T \bm{A}\bm{r}_k}{\bm{p}_k^T\bm{A}\bm{p}_k} \bm{p}_k. \end{equation*} !et !eblock @@ -726,26 +726,26 @@ This gives the following expression We can also compute the residual iteratively as !bt \begin{equation*} -\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, +\bm{r}_{k+1}=\bm{b}-\bm{A}\bm{x}_{k+1}, \end{equation*} !et which equals !bt \begin{equation*} -\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k), +\bm{b}-\bm{A}(\bm{x}_k+\alpha_k\bm{p}_k), \end{equation*} !et or !bt \begin{equation*} -(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k, +(\bm{b}-\bm{A}\bm{x}_k)-\alpha_k\bm{A}\bm{p}_k, \end{equation*} !et which gives !bt \begin{equation*} -\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k}, +\bm{r}_{k+1}=\bm{r}_k-\bm{A}\bm{p}_{k}, \end{equation*} !et !eblock @@ -776,22 +776,22 @@ with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally The linear regression model is given by !bt \[ -h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, +h_\beta(x) = \bm{y} = \beta_0 + \beta_1 x, \] !et such that !bt \[ -\hat{y}_i = \beta_0 + \beta_1 x_i. +\bm{y}_i = \beta_0 + \beta_1 x_i. \] !et !split ===== Gradient descent example ===== -Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$ +Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\bm{y}} = (\bm{y}_1,\cdots,\bm{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$ -It is convenient to write $\mathbf{\hat{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by +It is convenient to write $\mathbf{\bm{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by !bt \[ X \equiv \begin{bmatrix} @@ -827,7 +827,7 @@ where $X$ is the design matrix defined above. The Hessian matrix of $C(\beta)$ is given by !bt \[ -\hat{H} \equiv \begin{bmatrix} +\bm{H} \equiv \begin{bmatrix} \frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ \end{bmatrix} = 2X^T X. @@ -1447,9 +1447,9 @@ again understood to be element-wise operations below) \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\ \mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ \mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ -\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\ +\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ +\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \bm{\mathbf{m}}_t \over \sqrt{\bm{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \end{align} !et @@ -1460,13 +1460,13 @@ respectively, and $\eta$ and $\epsilon$ are identical to RMSprop. Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance -$\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - -(\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The +$\boldsymbol{\sigma}_t^2 = \bm{\mathbf{s}}_t - +(\bm{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The update rule for this parameter is given by !bt \[ -\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. +\Delta \theta_{t+1}= -\eta_t { \bm{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. \] !et