diff --git a/doc/pub/week37/html/._week37-bs022.html b/doc/pub/week37/html/._week37-bs022.html index 06891ca08..b8c55ebf6 100644 --- a/doc/pub/week37/html/._week37-bs022.html +++ b/doc/pub/week37/html/._week37-bs022.html @@ -339,8 +339,8 @@ function, which we want to minimize, can almost always be written as a sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),

$$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). +C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\theta}). $$ diff --git a/doc/pub/week37/html/._week37-bs023.html b/doc/pub/week37/html/._week37-bs023.html index dd4c9f5d6..deb1aa2a8 100644 --- a/doc/pub/week37/html/._week37-bs023.html +++ b/doc/pub/week37/html/._week37-bs023.html @@ -334,8 +334,8 @@ MathJax.Hub.Config({ computed as a sum over \( i \)-gradients

$$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). +\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}). $$

Stochasticity/randomness is introduced by only taking the diff --git a/doc/pub/week37/html/._week37-bs024.html b/doc/pub/week37/html/._week37-bs024.html index 56b9647ea..d20ff51bb 100644 --- a/doc/pub/week37/html/._week37-bs024.html +++ b/doc/pub/week37/html/._week37-bs024.html @@ -344,10 +344,10 @@ all data points with a sum over the data points in one the minibatches picked at random in each gradient descent step

$$ -\nabla_{\beta} -C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta -c_i(\mathbf{x}_i, \mathbf{\beta}). +\nabla_{\theta} +C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta +c_i(\mathbf{x}_i, \mathbf{\theta}). $$ diff --git a/doc/pub/week37/html/._week37-bs025.html b/doc/pub/week37/html/._week37-bs025.html index b29733b48..86a5e843e 100644 --- a/doc/pub/week37/html/._week37-bs025.html +++ b/doc/pub/week37/html/._week37-bs025.html @@ -332,8 +332,8 @@ MathJax.Hub.Config({

Thus a gradient descent step now looks like

$$ -\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) +\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) $$

where \( k \) is picked at random with equal diff --git a/doc/pub/week37/html/._week37-bs027.html b/doc/pub/week37/html/._week37-bs027.html index 3871aae9c..f2c24f1e7 100644 --- a/doc/pub/week37/html/._week37-bs027.html +++ b/doc/pub/week37/html/._week37-bs027.html @@ -338,7 +338,7 @@ is zero is valid also for local minima, so this would only tell us that we are close to a local/global minimum. However, we could also evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can -compare the values of the cost function and keep the \( \beta \) that +compare the values of the cost function and keep the \( \theta \) that gave the lowest value.

diff --git a/doc/pub/week37/html/._week37-bs029.html b/doc/pub/week37/html/._week37-bs029.html index cc383055d..702bebc0c 100644 --- a/doc/pub/week37/html/._week37-bs029.html +++ b/doc/pub/week37/html/._week37-bs029.html @@ -332,10 +332,10 @@ MathJax.Hub.Config({

As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).

-

In this way we can fix the number of epochs, compute \( \beta \) and +

In this way we can fix the number of epochs, compute \( \theta \) and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we -pick the final \( \beta \) that gives the lowest value of the cost +pick the final \( \theta \) that gives the lowest value of the cost function.

@@ -364,7 +364,7 @@ j = 0 for i in range(m): k = np.random.randint(m) #Pick the k-th minibatch at random #Compute the gradient using the data in minibatch Bk - #Compute new suggestion for beta + #Compute new suggestion for theta t = epoch*m+i gamma_j = step_length(t,t0,t1) j += 1 diff --git a/doc/pub/week37/html/._week37-bs041.html b/doc/pub/week37/html/._week37-bs041.html index c5bb38165..fc22b87ef 100644 --- a/doc/pub/week37/html/._week37-bs041.html +++ b/doc/pub/week37/html/._week37-bs041.html @@ -333,10 +333,10 @@ MathJax.Hub.Config({ Uses a decaying average of squared gradients (instead of a cumulative sum):

$$ -v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2, +v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2, $$ -

with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

+

with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

  1. Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).
  2. Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
    3. diff --git a/doc/pub/week37/html/._week37-bs043.html b/doc/pub/week37/html/._week37-bs043.html index 9bf87b4fe..79db666d2 100644 --- a/doc/pub/week37/html/._week37-bs043.html +++ b/doc/pub/week37/html/._week37-bs043.html @@ -340,13 +340,13 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \tag{3}\\ -\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\ +\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\ \boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber \end{align} $$ -

    where \( \beta \) controls the averaging time of the second moment and is -typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +

    where \( \theta \) controls the averaging time of the second moment and is +typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It diff --git a/doc/pub/week37/html/._week37-bs044.html b/doc/pub/week37/html/._week37-bs044.html index 68597f4df..0e1450067 100644 --- a/doc/pub/week37/html/._week37-bs044.html +++ b/doc/pub/week37/html/._week37-bs044.html @@ -355,16 +355,16 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \tag{4}\\ -\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ -\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\ +\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\ \boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \tag{5} \end{align} $$ -

    where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +

    where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

    diff --git a/doc/pub/week37/html/._week37-bs047.html b/doc/pub/week37/html/._week37-bs047.html index 05f8bec66..4bedba8b3 100644 --- a/doc/pub/week37/html/._week37-bs047.html +++ b/doc/pub/week37/html/._week37-bs047.html @@ -356,8 +356,8 @@ first example shows results with ordinary leats squares. import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) diff --git a/doc/pub/week37/html/._week37-bs048.html b/doc/pub/week37/html/._week37-bs048.html index f7a238e6a..d5074164e 100644 --- a/doc/pub/week37/html/._week37-bs048.html +++ b/doc/pub/week37/html/._week37-bs048.html @@ -343,8 +343,8 @@ MathJax.Hub.Config({ import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) diff --git a/doc/pub/week37/html/week37-reveal.html b/doc/pub/week37/html/week37-reveal.html index e4db185e2..572b0ee67 100644 --- a/doc/pub/week37/html/week37-reveal.html +++ b/doc/pub/week37/html/week37-reveal.html @@ -856,8 +856,8 @@ sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),

     
    $$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). +C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\theta}). $$

     
    @@ -870,8 +870,8 @@ computed as a sum over \( i \)-gradients

     
    $$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). +\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}). $$

     
    @@ -901,10 +901,10 @@ picked at random in each gradient descent step

     
    $$ -\nabla_{\beta} -C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta -c_i(\mathbf{x}_i, \mathbf{\beta}). +\nabla_{\theta} +C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta +c_i(\mathbf{x}_i, \mathbf{\theta}). $$

     
    @@ -915,8 +915,8 @@ $$

    Thus a gradient descent step now looks like

     
    $$ -\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) +\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) $$

     
    @@ -988,7 +988,7 @@ is zero is valid also for local minima, so this would only tell us that we are close to a local/global minimum. However, we could also evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can -compare the values of the cost function and keep the \( \beta \) that +compare the values of the cost function and keep the \( \theta \) that gave the lowest value.

    @@ -1015,10 +1015,10 @@ for a discussion of different scaling functions for the learning rate. $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$

     
    goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).

    -

    In this way we can fix the number of epochs, compute \( \beta \) and +

    In this way we can fix the number of epochs, compute \( \theta \) and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we -pick the final \( \beta \) that gives the lowest value of the cost +pick the final \( \theta \) that gives the lowest value of the cost function.

    @@ -1047,7 +1047,7 @@ j = 0 for i in range(m): k = np.random.randint(m) #Pick the k-th minibatch at random #Compute the gradient using the data in minibatch Bk - #Compute new suggestion for beta + #Compute new suggestion for theta t = epoch*m+i gamma_j = step_length(t,t0,t1) j += 1 @@ -1439,11 +1439,11 @@ Uses a decaying average of squared gradients (instead of a cumulative sum):

     
    $$ -v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2, +v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2, $$

     
    -

    with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

    +

    with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

    1. Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).

    2. Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
      3. @@ -1482,14 +1482,14 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \tag{3}\\ -\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\ +\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\ \boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber \end{align} $$

       
      -

      where \( \beta \) controls the averaging time of the second moment and is -typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +

      where \( \theta \) controls the averaging time of the second moment and is +typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It @@ -1529,17 +1529,17 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \tag{4}\\ -\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ -\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\ +\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\ \boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \tag{5} \end{align} $$

       
      -

      where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +

      where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

      @@ -1628,8 +1628,8 @@ first example shows results with ordinary leats squares. import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1702,8 +1702,8 @@ plt.show() import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1977,8 +1977,8 @@ delta_momentum = 0.3 import autograd.numpy as np from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1986,24 +1986,24 @@ y = 4+3* X = np.c_[np.ones((n,1)), x, x*x] XT_X = X.T @ X -beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) +theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) print("Own inversion") -print(beta_linreg) +print(theta_linreg) # Hessian matrix H = (2.0/n)* XT_X -# Note that here the Hessian does not depend on the parameters beta +# Note that here the Hessian does not depend on the parameters theta invH = np.linalg.pinv(H) -beta = np.random.randn(3,1) +theta = np.random.randn(3,1) Niterations = 5 # define the gradient training_gradient = grad(CostOLS) for iter in range(Niterations): - gradients = training_gradient(beta) - beta -= invH @ gradients + gradients = training_gradient(theta) + theta -= invH @ gradients print(iter,gradients[0],gradients[1]) -print("beta from own Newton code") -print(beta) +print("theta from own Newton code") +print(theta) @@ -2218,9 +2218,9 @@ theta = np.random.randn(3,# Value for learning rate eta = 0.01 -# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980 -beta1 = 0.9 -beta2 = 0.999 +# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980 +theta1 = 0.9 +theta2 = 0.999 # Including AdaGrad parameter to avoid possible division by zero delta = 1e-7 iter = 0 @@ -2234,10 +2234,10 @@ delta = 1e-7 yi = y[random_index:random_index+M] gradients = (1.0/M)*training_gradient(yi, xi, theta) # Computing moments first - first_moment = beta1*first_moment + (1-beta1)*gradients - second_moment = beta2*second_moment+(1-beta2)*gradients*gradients - first_term = first_moment/(1.0-beta1**iter) - second_term = second_moment/(1.0-beta2**iter) + first_moment = theta1*first_moment + (1-theta1)*gradients + second_moment = theta2*second_moment+(1-theta2)*gradients*gradients + first_term = first_moment/(1.0-theta1**iter) + second_term = second_moment/(1.0-theta2**iter) # Scaling with rho the new and the previous results update = eta*first_term/(np.sqrt(second_term)+delta) theta -= update @@ -2269,12 +2269,9 @@ delta = 1e-7

    3. Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)

    4. Work on project 1
      5. - -

    +

    +

    For more discussions of Ridge regression and calculation of averages, Wessel van Wieringen's article is highly recommended.

    diff --git a/doc/pub/week37/html/week37-solarized.html b/doc/pub/week37/html/week37-solarized.html index 381b60233..daf6c2255 100644 --- a/doc/pub/week37/html/week37-solarized.html +++ b/doc/pub/week37/html/week37-solarized.html @@ -909,8 +909,8 @@ function, which we want to minimize, can almost always be written as a sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),

    $$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). +C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\theta}). $$ @@ -921,8 +921,8 @@ $$ computed as a sum over \( i \)-gradients

    $$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). +\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}). $$

    Stochasticity/randomness is introduced by only taking the @@ -949,10 +949,10 @@ all data points with a sum over the data points in one the minibatches picked at random in each gradient descent step

    $$ -\nabla_{\beta} -C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta -c_i(\mathbf{x}_i, \mathbf{\beta}). +\nabla_{\theta} +C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta +c_i(\mathbf{x}_i, \mathbf{\theta}). $$ @@ -961,8 +961,8 @@ $$

    Thus a gradient descent step now looks like

    $$ -\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) +\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) $$

    where \( k \) is picked at random with equal @@ -1031,7 +1031,7 @@ is zero is valid also for local minima, so this would only tell us that we are close to a local/global minimum. However, we could also evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can -compare the values of the cost function and keep the \( \beta \) that +compare the values of the cost function and keep the \( \theta \) that gave the lowest value.

    @@ -1054,10 +1054,10 @@ for a discussion of different scaling functions for the learning rate.

    As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).

    -

    In this way we can fix the number of epochs, compute \( \beta \) and +

    In this way we can fix the number of epochs, compute \( \theta \) and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we -pick the final \( \beta \) that gives the lowest value of the cost +pick the final \( \theta \) that gives the lowest value of the cost function.

    @@ -1086,7 +1086,7 @@ j = 0 for i in range(m): k = np.random.randint(m) #Pick the k-th minibatch at random #Compute the gradient using the data in minibatch Bk - #Compute new suggestion for beta + #Compute new suggestion for theta t = epoch*m+i gamma_j = step_length(t,t0,t1) j += 1 @@ -1441,10 +1441,10 @@ $$ Uses a decaying average of squared gradients (instead of a cumulative sum):

    $$ -v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2, +v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2, $$ -

    with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

    +

    with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

    1. Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).
    2. Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
      3. @@ -1478,13 +1478,13 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \label{_auto3}\\ -\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\ +\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\ \boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber \end{align} $$ -

      where \( \beta \) controls the averaging time of the second moment and is -typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +

      where \( \theta \) controls the averaging time of the second moment and is +typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It @@ -1522,16 +1522,16 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \label{_auto4}\\ -\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ -\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\ +\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\ \boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \label{_auto5} \end{align} $$ -

      where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +

      where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

      @@ -1615,8 +1615,8 @@ first example shows results with ordinary leats squares. import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1689,8 +1689,8 @@ plt.show() import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1964,8 +1964,8 @@ delta_momentum = 0.3 import autograd.numpy as np from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1973,24 +1973,24 @@ y = 4+3* X = np.c_[np.ones((n,1)), x, x*x] XT_X = X.T @ X -beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) +theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) print("Own inversion") -print(beta_linreg) +print(theta_linreg) # Hessian matrix H = (2.0/n)* XT_X -# Note that here the Hessian does not depend on the parameters beta +# Note that here the Hessian does not depend on the parameters theta invH = np.linalg.pinv(H) -beta = np.random.randn(3,1) +theta = np.random.randn(3,1) Niterations = 5 # define the gradient training_gradient = grad(CostOLS) for iter in range(Niterations): - gradients = training_gradient(beta) - beta -= invH @ gradients + gradients = training_gradient(theta) + theta -= invH @ gradients print(iter,gradients[0],gradients[1]) -print("beta from own Newton code") -print(beta) +print("theta from own Newton code") +print(theta) @@ -2204,9 +2204,9 @@ theta = np.random.randn(3,# Value for learning rate eta = 0.01 -# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980 -beta1 = 0.9 -beta2 = 0.999 +# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980 +theta1 = 0.9 +theta2 = 0.999 # Including AdaGrad parameter to avoid possible division by zero delta = 1e-7 iter = 0 @@ -2220,10 +2220,10 @@ delta = 1e-7 yi = y[random_index:random_index+M] gradients = (1.0/M)*training_gradient(yi, xi, theta) # Computing moments first - first_moment = beta1*first_moment + (1-beta1)*gradients - second_moment = beta2*second_moment+(1-beta2)*gradients*gradients - first_term = first_moment/(1.0-beta1**iter) - second_term = second_moment/(1.0-beta2**iter) + first_moment = theta1*first_moment + (1-theta1)*gradients + second_moment = theta2*second_moment+(1-theta2)*gradients*gradients + first_term = first_moment/(1.0-theta1**iter) + second_term = second_moment/(1.0-theta2**iter) # Scaling with rho the new and the previous results update = eta*first_term/(np.sqrt(second_term)+delta) theta -= update @@ -2255,10 +2255,8 @@ delta = 1e-7
    3. Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)
    4. Work on project 1
      5. -
    +

    For more discussions of Ridge regression and calculation of averages, Wessel van Wieringen's article is highly recommended.

    diff --git a/doc/pub/week37/html/week37.html b/doc/pub/week37/html/week37.html index 91bd25e27..445adef5a 100644 --- a/doc/pub/week37/html/week37.html +++ b/doc/pub/week37/html/week37.html @@ -986,8 +986,8 @@ function, which we want to minimize, can almost always be written as a sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),

    $$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). +C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\theta}). $$ @@ -998,8 +998,8 @@ $$ computed as a sum over \( i \)-gradients

    $$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). +\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}). $$

    Stochasticity/randomness is introduced by only taking the @@ -1026,10 +1026,10 @@ all data points with a sum over the data points in one the minibatches picked at random in each gradient descent step

    $$ -\nabla_{\beta} -C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta -c_i(\mathbf{x}_i, \mathbf{\beta}). +\nabla_{\theta} +C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta +c_i(\mathbf{x}_i, \mathbf{\theta}). $$ @@ -1038,8 +1038,8 @@ $$

    Thus a gradient descent step now looks like

    $$ -\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) +\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) $$

    where \( k \) is picked at random with equal @@ -1108,7 +1108,7 @@ is zero is valid also for local minima, so this would only tell us that we are close to a local/global minimum. However, we could also evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can -compare the values of the cost function and keep the \( \beta \) that +compare the values of the cost function and keep the \( \theta \) that gave the lowest value.

    @@ -1131,10 +1131,10 @@ for a discussion of different scaling functions for the learning rate.

    As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).

    -

    In this way we can fix the number of epochs, compute \( \beta \) and +

    In this way we can fix the number of epochs, compute \( \theta \) and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we -pick the final \( \beta \) that gives the lowest value of the cost +pick the final \( \theta \) that gives the lowest value of the cost function.

    @@ -1163,7 +1163,7 @@ j = 0 for i in range(m): k = np.random.randint(m) #Pick the k-th minibatch at random #Compute the gradient using the data in minibatch Bk - #Compute new suggestion for beta + #Compute new suggestion for theta t = epoch*m+i gamma_j = step_length(t,t0,t1) j += 1 @@ -1518,10 +1518,10 @@ $$ Uses a decaying average of squared gradients (instead of a cumulative sum):

    $$ -v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2, +v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2, $$ -

    with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

    +

    with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).

    1. Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).
    2. Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
      3. @@ -1555,13 +1555,13 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \label{_auto3}\\ -\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\ +\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\ \boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber \end{align} $$ -

      where \( \beta \) controls the averaging time of the second moment and is -typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +

      where \( \theta \) controls the averaging time of the second moment and is +typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It @@ -1599,16 +1599,16 @@ $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \label{_auto4}\\ -\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ -\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\ +\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\ +\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\ +\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\ \boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \label{_auto5} \end{align} $$ -

      where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +

      where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

      @@ -1692,8 +1692,8 @@ first example shows results with ordinary leats squares. import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1766,8 +1766,8 @@ plt.show() import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -2041,8 +2041,8 @@ delta_momentum = import autograd.numpy as np from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -2050,24 +2050,24 @@ y = 4+3* X = np.c_[np.ones((n,1)), x, x*x] XT_X = X.T @ X -beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) +theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) print("Own inversion") -print(beta_linreg) +print(theta_linreg) # Hessian matrix H = (2.0/n)* XT_X -# Note that here the Hessian does not depend on the parameters beta +# Note that here the Hessian does not depend on the parameters theta invH = np.linalg.pinv(H) -beta = np.random.randn(3,1) +theta = np.random.randn(3,1) Niterations = 5 # define the gradient training_gradient = grad(CostOLS) for iter in range(Niterations): - gradients = training_gradient(beta) - beta -= invH @ gradients + gradients = training_gradient(theta) + theta -= invH @ gradients print(iter,gradients[0],gradients[1]) -print("beta from own Newton code") -print(beta) +print("theta from own Newton code") +print(theta) @@ -2281,9 +2281,9 @@ theta = np.# Value for learning rate eta = 0.01 -# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980 -beta1 = 0.9 -beta2 = 0.999 +# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980 +theta1 = 0.9 +theta2 = 0.999 # Including AdaGrad parameter to avoid possible division by zero delta = 1e-7 iter = 0 @@ -2297,10 +2297,10 @@ delta = 1e-7= y[random_index:random_index+M] gradients = (1.0/M)*training_gradient(yi, xi, theta) # Computing moments first - first_moment = beta1*first_moment + (1-beta1)*gradients - second_moment = beta2*second_moment+(1-beta2)*gradients*gradients - first_term = first_moment/(1.0-beta1**iter) - second_term = second_moment/(1.0-beta2**iter) + first_moment = theta1*first_moment + (1-theta1)*gradients + second_moment = theta2*second_moment+(1-theta2)*gradients*gradients + first_term = first_moment/(1.0-theta1**iter) + second_term = second_moment/(1.0-theta2**iter) # Scaling with rho the new and the previous results update = eta*first_term/(np.sqrt(second_term)+delta) theta -= update @@ -2332,10 +2332,8 @@ delta = 1e-7 Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)
    3. Work on project 1
      4. -
        -
      • For more discussions of Ridge regression and calculation of averages, Wessel van Wieringen's article is highly recommended.
        • -
    +

    For more discussions of Ridge regression and calculation of averages, Wessel van Wieringen's article is highly recommended.

    diff --git a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz index 42eedc60a..cc9e97db7 100644 Binary files a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz and b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz differ diff --git a/doc/pub/week37/ipynb/week37.ipynb b/doc/pub/week37/ipynb/week37.ipynb index acd6b4e52..24c7e599e 100644 --- a/doc/pub/week37/ipynb/week37.ipynb +++ b/doc/pub/week37/ipynb/week37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "c543a71a", + "id": "12943921", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "4ff6b050", + "id": "ab900606", "metadata": { "editable": true }, @@ -29,7 +29,7 @@ }, { "cell_type": "markdown", - "id": "a7079d3e", + "id": "5b82a2e8", "metadata": { "editable": true }, @@ -52,7 +52,7 @@ }, { "cell_type": "markdown", - "id": "db3af8cc", + "id": "ae219e88", "metadata": { "editable": true }, @@ -69,7 +69,7 @@ }, { "cell_type": "markdown", - "id": "79d6d5be", + "id": "f6f72b45", "metadata": { "editable": true }, @@ -79,7 +79,7 @@ }, { "cell_type": "markdown", - "id": "fa427fc3", + "id": "760ee28d", "metadata": { "editable": true }, @@ -103,7 +103,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "1185629e", + "id": "86fc694c", "metadata": { "collapsed": false, "editable": true @@ -116,7 +116,7 @@ }, { "cell_type": "markdown", - "id": "42dfe293", + "id": "20296a13", "metadata": { "editable": true }, @@ -127,7 +127,7 @@ }, { "cell_type": "markdown", - "id": "a726cc6e", + "id": "cd2ed57d", "metadata": { "editable": true }, @@ -139,7 +139,7 @@ }, { "cell_type": "markdown", - "id": "991c8919", + "id": "09f879d6", "metadata": { "editable": true }, @@ -149,7 +149,7 @@ }, { "cell_type": "markdown", - "id": "ba895122", + "id": "4c1e3d77", "metadata": { "editable": true }, @@ -161,7 +161,7 @@ }, { "cell_type": "markdown", - "id": "5c2b9317", + "id": "81797ec3", "metadata": { "editable": true }, @@ -175,7 +175,7 @@ }, { "cell_type": "markdown", - "id": "264eef67", + "id": "2ddb2644", "metadata": { "editable": true }, @@ -191,7 +191,7 @@ }, { "cell_type": "markdown", - "id": "a451a9ce", + "id": "16e40780", "metadata": { "editable": true }, @@ -201,7 +201,7 @@ }, { "cell_type": "markdown", - "id": "e7fa569b", + "id": "eea52a64", "metadata": { "editable": true }, @@ -213,7 +213,7 @@ }, { "cell_type": "markdown", - "id": "a2655bc6", + "id": "b6bcc86e", "metadata": { "editable": true }, @@ -223,7 +223,7 @@ }, { "cell_type": "markdown", - "id": "277ace2e", + "id": "c4efac19", "metadata": { "editable": true }, @@ -235,7 +235,7 @@ }, { "cell_type": "markdown", - "id": "75bbf03c", + "id": "3e9ab34e", "metadata": { "editable": true }, @@ -249,7 +249,7 @@ }, { "cell_type": "markdown", - "id": "d5bb2db2", + "id": "96fdc0e2", "metadata": { "editable": true }, @@ -259,7 +259,7 @@ }, { "cell_type": "markdown", - "id": "53430d47", + "id": "9667066b", "metadata": { "editable": true }, @@ -270,7 +270,7 @@ }, { "cell_type": "markdown", - "id": "89dce328", + "id": "d83b3d29", "metadata": { "editable": true }, @@ -285,7 +285,7 @@ }, { "cell_type": "markdown", - "id": "542b8278", + "id": "f886d5ed", "metadata": { "editable": true }, @@ -295,7 +295,7 @@ }, { "cell_type": "markdown", - "id": "2b94baca", + "id": "eba60cb9", "metadata": { "editable": true }, @@ -307,7 +307,7 @@ }, { "cell_type": "markdown", - "id": "bfc6b93f", + "id": "0bc7b9a7", "metadata": { "editable": true }, @@ -319,7 +319,7 @@ }, { "cell_type": "markdown", - "id": "703de92e", + "id": "1c762420", "metadata": { "editable": true }, @@ -334,7 +334,7 @@ }, { "cell_type": "markdown", - "id": "f843aae1", + "id": "1417bf2c", "metadata": { "editable": true }, @@ -347,7 +347,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "b76443c9", + "id": "9df5f8e0", "metadata": { "collapsed": false, "editable": true @@ -406,7 +406,7 @@ }, { "cell_type": "markdown", - "id": "7b75f4ec", + "id": "13e52a61", "metadata": { "editable": true }, @@ -418,7 +418,7 @@ }, { "cell_type": "markdown", - "id": "47ecdee7", + "id": "a0c3aad5", "metadata": { "editable": true }, @@ -430,7 +430,7 @@ }, { "cell_type": "markdown", - "id": "ff25299b", + "id": "e2cdcdf4", "metadata": { "editable": true }, @@ -440,7 +440,7 @@ }, { "cell_type": "markdown", - "id": "21a88a84", + "id": "48448ba8", "metadata": { "editable": true }, @@ -454,7 +454,7 @@ }, { "cell_type": "markdown", - "id": "67491c15", + "id": "f9ba7343", "metadata": { "editable": true }, @@ -464,7 +464,7 @@ }, { "cell_type": "markdown", - "id": "f1ac13f1", + "id": "061c400a", "metadata": { "editable": true }, @@ -476,7 +476,7 @@ }, { "cell_type": "markdown", - "id": "67d83802", + "id": "991d1578", "metadata": { "editable": true }, @@ -487,7 +487,7 @@ }, { "cell_type": "markdown", - "id": "75935d70", + "id": "ed8e52e3", "metadata": { "editable": true }, @@ -502,7 +502,7 @@ }, { "cell_type": "markdown", - "id": "4707a65d", + "id": "324558f2", "metadata": { "editable": true }, @@ -516,7 +516,7 @@ }, { "cell_type": "markdown", - "id": "ab16b163", + "id": "4b3bafd8", "metadata": { "editable": true }, @@ -527,7 +527,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "fe7ee1f7", + "id": "66c8efc3", "metadata": { "collapsed": false, "editable": true @@ -588,7 +588,7 @@ }, { "cell_type": "markdown", - "id": "823fdaf7", + "id": "6aa8fcba", "metadata": { "editable": true }, @@ -610,7 +610,7 @@ }, { "cell_type": "markdown", - "id": "21ea3f2a", + "id": "60f264ef", "metadata": { "editable": true }, @@ -623,7 +623,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "5b729a7e", + "id": "7ea138ee", "metadata": { "collapsed": false, "editable": true @@ -689,7 +689,7 @@ }, { "cell_type": "markdown", - "id": "209fc69c", + "id": "463fe2a0", "metadata": { "editable": true }, @@ -700,7 +700,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "984c4202", + "id": "cd5466dd", "metadata": { "collapsed": false, "editable": true @@ -774,7 +774,7 @@ }, { "cell_type": "markdown", - "id": "8577b238", + "id": "dc904f58", "metadata": { "editable": true }, @@ -793,7 +793,7 @@ }, { "cell_type": "markdown", - "id": "c11d0874", + "id": "26e80e0e", "metadata": { "editable": true }, @@ -814,7 +814,7 @@ }, { "cell_type": "markdown", - "id": "64ad52d0", + "id": "b6675474", "metadata": { "editable": true }, @@ -830,7 +830,7 @@ }, { "cell_type": "markdown", - "id": "5dd4c467", + "id": "a37f21c1", "metadata": { "editable": true }, @@ -844,7 +844,7 @@ }, { "cell_type": "markdown", - "id": "42904823", + "id": "d109bed4", "metadata": { "editable": true }, @@ -872,7 +872,7 @@ }, { "cell_type": "markdown", - "id": "49dfaf2e", + "id": "9fb274a9", "metadata": { "editable": true }, @@ -904,7 +904,7 @@ }, { "cell_type": "markdown", - "id": "284b8663", + "id": "eff581f1", "metadata": { "editable": true }, @@ -921,20 +921,20 @@ }, { "cell_type": "markdown", - "id": "5e8cc1b8", + "id": "1aff7a14", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "510505a0", + "id": "74b749c5", "metadata": { "editable": true }, @@ -947,20 +947,20 @@ }, { "cell_type": "markdown", - "id": "32fc501f", + "id": "32f5e586", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "\\nabla_\\theta C(\\mathbf{\\theta}) = \\sum_i^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "9dcac7ef", + "id": "3ab67fbd", "metadata": { "editable": true }, @@ -974,7 +974,7 @@ }, { "cell_type": "markdown", - "id": "c9464120", + "id": "bc199cc8", "metadata": { "editable": true }, @@ -996,22 +996,22 @@ }, { "cell_type": "markdown", - "id": "59bc847c", + "id": "232109bf", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_{\\beta}\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\beta\n", - "c_i(\\mathbf{x}_i, \\mathbf{\\beta}).\n", + "\\nabla_{\\theta}\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\theta\n", + "c_i(\\mathbf{x}_i, \\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "ba220299", + "id": "d6870f3a", "metadata": { "editable": true }, @@ -1023,20 +1023,20 @@ }, { "cell_type": "markdown", - "id": "8608ba34", + "id": "2345af83", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta})\n", + "\\theta_{j+1} = \\theta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta})\n", "$$" ] }, { "cell_type": "markdown", - "id": "0faa1782", + "id": "3754bc41", "metadata": { "editable": true }, @@ -1050,7 +1050,7 @@ }, { "cell_type": "markdown", - "id": "97eddb68", + "id": "b9f2997a", "metadata": { "editable": true }, @@ -1061,7 +1061,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "309da17e", + "id": "8bfcd7e3", "metadata": { "collapsed": false, "editable": true @@ -1086,7 +1086,7 @@ }, { "cell_type": "markdown", - "id": "7aea4344", + "id": "10d0b55b", "metadata": { "editable": true }, @@ -1102,7 +1102,7 @@ }, { "cell_type": "markdown", - "id": "52f8d549", + "id": "fb3b2039", "metadata": { "editable": true }, @@ -1117,13 +1117,13 @@ "that we are close to a local/global minimum. However, we could also\n", "evaluate the cost function at this point, store the result and\n", "continue the search. If the test kicks in at a later stage we can\n", - "compare the values of the cost function and keep the $\\beta$ that\n", + "compare the values of the cost function and keep the $\\theta$ that\n", "gave the lowest value." ] }, { "cell_type": "markdown", - "id": "dcd04de7", + "id": "941224bf", "metadata": { "editable": true }, @@ -1143,7 +1143,7 @@ }, { "cell_type": "markdown", - "id": "4ecf0b16", + "id": "1b3481bb", "metadata": { "editable": true }, @@ -1152,17 +1152,17 @@ "\n", "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n", "\n", - "In this way we can fix the number of epochs, compute $\\beta$ and\n", + "In this way we can fix the number of epochs, compute $\\theta$ and\n", "evaluate the cost function at the end. Repeating the computation will\n", "give a different result since the scheme is random by design. Then we\n", - "pick the final $\\beta$ that gives the lowest value of the cost\n", + "pick the final $\\theta$ that gives the lowest value of the cost\n", "function." ] }, { "cell_type": "code", "execution_count": 7, - "id": "71eb89db", + "id": "f36e40ff", "metadata": { "collapsed": false, "editable": true @@ -1187,7 +1187,7 @@ " for i in range(m):\n", " k = np.random.randint(m) #Pick the k-th minibatch at random\n", " #Compute the gradient using the data in minibatch Bk\n", - " #Compute new suggestion for beta\n", + " #Compute new suggestion for theta\n", " t = epoch*m+i\n", " gamma_j = step_length(t,t0,t1)\n", " j += 1\n", @@ -1197,7 +1197,7 @@ }, { "cell_type": "markdown", - "id": "10ce3fc7", + "id": "ae32760a", "metadata": { "editable": true }, @@ -1210,7 +1210,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "f2d402a8", + "id": "d374888d", "metadata": { "collapsed": false, "editable": true @@ -1287,7 +1287,7 @@ }, { "cell_type": "markdown", - "id": "ffd1e92d", + "id": "4b7b2fd0", "metadata": { "editable": true }, @@ -1302,7 +1302,7 @@ }, { "cell_type": "markdown", - "id": "4b72d3f9", + "id": "b179c428", "metadata": { "editable": true }, @@ -1317,7 +1317,7 @@ }, { "cell_type": "markdown", - "id": "1b60ec02", + "id": "ee4f36ae", "metadata": { "editable": true }, @@ -1329,7 +1329,7 @@ }, { "cell_type": "markdown", - "id": "e1fa24ba", + "id": "52ffa296", "metadata": { "editable": true }, @@ -1347,7 +1347,7 @@ }, { "cell_type": "markdown", - "id": "7a1fc9fc", + "id": "358f57fc", "metadata": { "editable": true }, @@ -1366,7 +1366,7 @@ }, { "cell_type": "markdown", - "id": "23abda5d", + "id": "98758ad1", "metadata": { "editable": true }, @@ -1378,7 +1378,7 @@ }, { "cell_type": "markdown", - "id": "0f0067a9", + "id": "dc9c06a6", "metadata": { "editable": true }, @@ -1388,7 +1388,7 @@ }, { "cell_type": "markdown", - "id": "b86ec959", + "id": "9f4be6e8", "metadata": { "editable": true }, @@ -1404,7 +1404,7 @@ }, { "cell_type": "markdown", - "id": "7685ff97", + "id": "2cada246", "metadata": { "editable": true }, @@ -1416,7 +1416,7 @@ }, { "cell_type": "markdown", - "id": "28203c1e", + "id": "1ad6f530", "metadata": { "editable": true }, @@ -1426,7 +1426,7 @@ }, { "cell_type": "markdown", - "id": "1495b23f", + "id": "a366b9e1", "metadata": { "editable": true }, @@ -1438,7 +1438,7 @@ }, { "cell_type": "markdown", - "id": "15bbc6e6", + "id": "f993aacc", "metadata": { "editable": true }, @@ -1448,7 +1448,7 @@ }, { "cell_type": "markdown", - "id": "ffeffc05", + "id": "40c1c570", "metadata": { "editable": true }, @@ -1460,7 +1460,7 @@ }, { "cell_type": "markdown", - "id": "166cfafc", + "id": "ec3b1a6a", "metadata": { "editable": true }, @@ -1476,7 +1476,7 @@ }, { "cell_type": "markdown", - "id": "486fadf0", + "id": "af2ed825", "metadata": { "editable": true }, @@ -1488,7 +1488,7 @@ }, { "cell_type": "markdown", - "id": "cdb0769d", + "id": "dcb9112b", "metadata": { "editable": true }, @@ -1521,7 +1521,7 @@ }, { "cell_type": "markdown", - "id": "e65569bb", + "id": "e046df57", "metadata": { "editable": true }, @@ -1533,7 +1533,7 @@ }, { "cell_type": "markdown", - "id": "9d72d316", + "id": "39adf18c", "metadata": { "editable": true }, @@ -1551,7 +1551,7 @@ }, { "cell_type": "markdown", - "id": "b7b9811e", + "id": "99c86b52", "metadata": { "editable": true }, @@ -1562,7 +1562,7 @@ }, { "cell_type": "markdown", - "id": "d8f3c523", + "id": "d70efd9b", "metadata": { "editable": true }, @@ -1593,7 +1593,7 @@ }, { "cell_type": "markdown", - "id": "22d6be74", + "id": "165c32ca", "metadata": { "editable": true }, @@ -1615,7 +1615,7 @@ }, { "cell_type": "markdown", - "id": "091ace5a", + "id": "8abe000c", "metadata": { "editable": true }, @@ -1635,7 +1635,7 @@ }, { "cell_type": "markdown", - "id": "5c05fb8b", + "id": "06880daa", "metadata": { "editable": true }, @@ -1655,7 +1655,7 @@ }, { "cell_type": "markdown", - "id": "21fb8bd0", + "id": "bd52ca3b", "metadata": { "editable": true }, @@ -1667,7 +1667,7 @@ }, { "cell_type": "markdown", - "id": "f71dfafa", + "id": "b60d88ef", "metadata": { "editable": true }, @@ -1679,7 +1679,7 @@ }, { "cell_type": "markdown", - "id": "64869e2e", + "id": "f193d2fc", "metadata": { "editable": true }, @@ -1691,7 +1691,7 @@ }, { "cell_type": "markdown", - "id": "c2d8d6b1", + "id": "b5906876", "metadata": { "editable": true }, @@ -1703,7 +1703,7 @@ }, { "cell_type": "markdown", - "id": "2171198a", + "id": "51c89de1", "metadata": { "editable": true }, @@ -1714,7 +1714,7 @@ }, { "cell_type": "markdown", - "id": "353e21d3", + "id": "7657234b", "metadata": { "editable": true }, @@ -1726,7 +1726,7 @@ }, { "cell_type": "markdown", - "id": "b59e6add", + "id": "ca84e62b", "metadata": { "editable": true }, @@ -1736,7 +1736,7 @@ }, { "cell_type": "markdown", - "id": "707caefe", + "id": "f2811ae9", "metadata": { "editable": true }, @@ -1748,7 +1748,7 @@ }, { "cell_type": "markdown", - "id": "62dafce4", + "id": "1ac363c6", "metadata": { "editable": true }, @@ -1758,7 +1758,7 @@ }, { "cell_type": "markdown", - "id": "01a2ba61", + "id": "50e6f287", "metadata": { "editable": true }, @@ -1778,7 +1778,7 @@ }, { "cell_type": "markdown", - "id": "b4bc88e0", + "id": "9bd4eb6c", "metadata": { "editable": true }, @@ -1790,24 +1790,24 @@ }, { "cell_type": "markdown", - "id": "4933fddb", + "id": "02366006", "metadata": { "editable": true }, "source": [ "$$\n", - "v_t = \\beta_2\\, v_{t-1} + (1-\\beta_2)\\, (\\nabla L(w_t))^2,\n", + "v_t = \\theta_2\\, v_{t-1} + (1-\\theta_2)\\, (\\nabla L(w_t))^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "0dd8edf7", + "id": "7913c18d", "metadata": { "editable": true }, "source": [ - "with $\\beta_2$ typically $0.9$ (or $0.99$).\n", + "with $\\theta_2$ typically $0.9$ (or $0.99$).\n", "1. Update: $w_{t+1} = w_t - \\frac{\\alpha}{\\sqrt{v_t + \\epsilon}} \\nabla L(w_t)$.\n", "\n", "2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n", @@ -1819,7 +1819,7 @@ }, { "cell_type": "markdown", - "id": "1ce98f28", + "id": "2374dc95", "metadata": { "editable": true }, @@ -1841,7 +1841,7 @@ }, { "cell_type": "markdown", - "id": "d7d646bc", + "id": "e6ec6c1d", "metadata": { "editable": true }, @@ -1856,7 +1856,7 @@ }, { "cell_type": "markdown", - "id": "3a212292", + "id": "e7bd907d", "metadata": { "editable": true }, @@ -1874,19 +1874,19 @@ }, { "cell_type": "markdown", - "id": "88c0ec96", + "id": "2b52a891", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", + "\\mathbf{s}_t =\\theta \\mathbf{s}_{t-1} +(1-\\theta)\\mathbf{g}_t^2 \\nonumber\n", "$$" ] }, { "cell_type": "markdown", - "id": "9cfb794f", + "id": "a1cba407", "metadata": { "editable": true }, @@ -1898,13 +1898,13 @@ }, { "cell_type": "markdown", - "id": "aaa4936c", + "id": "56d45945", "metadata": { "editable": true }, "source": [ - "where $\\beta$ controls the averaging time of the second moment and is\n", - "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", + "where $\\theta$ controls the averaging time of the second moment and is\n", + "typically taken to be about $\\theta=0.9$, $\\eta_t$ is a learning rate\n", "typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a\n", "small regularization constant to prevent divergences. Multiplication\n", "and division by vectors is understood as an element-wise operation. It\n", @@ -1916,7 +1916,7 @@ }, { "cell_type": "markdown", - "id": "5493a6cd", + "id": "624a4249", "metadata": { "editable": true }, @@ -1945,7 +1945,7 @@ }, { "cell_type": "markdown", - "id": "35795070", + "id": "01d10d3b", "metadata": { "editable": true }, @@ -1963,55 +1963,55 @@ }, { "cell_type": "markdown", - "id": "1e67e6dd", + "id": "be9127fc", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", + "\\mathbf{m}_t = \\theta_1 \\mathbf{m}_{t-1} + (1-\\theta_1) \\mathbf{g}_t \\nonumber\n", "$$" ] }, { "cell_type": "markdown", - "id": "39d582bb", + "id": "9530fa88", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", + "\\mathbf{s}_t =\\theta_2 \\mathbf{s}_{t-1} +(1-\\theta_2)\\mathbf{g}_t^2 \\nonumber\n", "$$" ] }, { "cell_type": "markdown", - "id": "4a7e2423", + "id": "ff70dd61", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", + "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\theta_1^t} \\nonumber\n", "$$" ] }, { "cell_type": "markdown", - "id": "53da77c2", + "id": "b60aa26c", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", + "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\theta_2^t} \\nonumber\n", "$$" ] }, { "cell_type": "markdown", - "id": "a4c7df45", + "id": "1d9fd822", "metadata": { "editable": true }, @@ -2023,7 +2023,7 @@ }, { "cell_type": "markdown", - "id": "f645a0aa", + "id": "ac5ec088", "metadata": { "editable": true }, @@ -2040,12 +2040,12 @@ }, { "cell_type": "markdown", - "id": "505619bb", + "id": "98304a21", "metadata": { "editable": true }, "source": [ - "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", + "where $\\theta_1$ and $\\theta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", "respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n", "\n", @@ -2059,7 +2059,7 @@ }, { "cell_type": "markdown", - "id": "6cb4c9a9", + "id": "23bbb8a1", "metadata": { "editable": true }, @@ -2071,7 +2071,7 @@ }, { "cell_type": "markdown", - "id": "39627950", + "id": "7c742e61", "metadata": { "editable": true }, @@ -2085,7 +2085,7 @@ }, { "cell_type": "markdown", - "id": "f240057d", + "id": "4b47969f", "metadata": { "editable": true }, @@ -2101,7 +2101,7 @@ }, { "cell_type": "markdown", - "id": "428ee09d", + "id": "d9de6d80", "metadata": { "editable": true }, @@ -2117,7 +2117,7 @@ }, { "cell_type": "markdown", - "id": "9ce3cc3e", + "id": "fb562f5c", "metadata": { "editable": true }, @@ -2133,7 +2133,7 @@ }, { "cell_type": "markdown", - "id": "193b3778", + "id": "37254cef", "metadata": { "editable": true }, @@ -2151,7 +2151,7 @@ }, { "cell_type": "markdown", - "id": "8d7187f5", + "id": "ebb970a6", "metadata": { "editable": true }, @@ -2172,7 +2172,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "68d35b76", + "id": "33165d6b", "metadata": { "collapsed": false, "editable": true @@ -2186,8 +2186,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2232,7 +2232,7 @@ }, { "cell_type": "markdown", - "id": "f39b471c", + "id": "8d49a811", "metadata": { "editable": true }, @@ -2243,7 +2243,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "265059ef", + "id": "733f4d02", "metadata": { "collapsed": false, "editable": true @@ -2257,8 +2257,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2307,7 +2307,7 @@ }, { "cell_type": "markdown", - "id": "18183fb0", + "id": "b0c40b1f", "metadata": { "editable": true }, @@ -2322,7 +2322,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "4d9d93b3", + "id": "0484f076", "metadata": { "collapsed": false, "editable": true @@ -2406,7 +2406,7 @@ }, { "cell_type": "markdown", - "id": "1652a471", + "id": "be0eb333", "metadata": { "editable": true }, @@ -2417,7 +2417,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "6092712c", + "id": "430ff8e3", "metadata": { "collapsed": false, "editable": true @@ -2495,7 +2495,7 @@ }, { "cell_type": "markdown", - "id": "4d2f230d", + "id": "9df5b099", "metadata": { "editable": true }, @@ -2508,7 +2508,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "b46cedcc", + "id": "14fd48f7", "metadata": { "collapsed": false, "editable": true @@ -2521,8 +2521,8 @@ "import autograd.numpy as np\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2530,29 +2530,29 @@ "\n", "X = np.c_[np.ones((n,1)), x, x*x]\n", "XT_X = X.T @ X\n", - "beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", + "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", "print(\"Own inversion\")\n", - "print(beta_linreg)\n", + "print(theta_linreg)\n", "# Hessian matrix\n", "H = (2.0/n)* XT_X\n", - "# Note that here the Hessian does not depend on the parameters beta\n", + "# Note that here the Hessian does not depend on the parameters theta\n", "invH = np.linalg.pinv(H)\n", - "beta = np.random.randn(3,1)\n", + "theta = np.random.randn(3,1)\n", "Niterations = 5\n", "# define the gradient\n", "training_gradient = grad(CostOLS)\n", "\n", "for iter in range(Niterations):\n", - " gradients = training_gradient(beta)\n", - " beta -= invH @ gradients\n", + " gradients = training_gradient(theta)\n", + " theta -= invH @ gradients\n", " print(iter,gradients[0],gradients[1])\n", - "print(\"beta from own Newton code\")\n", - "print(beta)" + "print(\"theta from own Newton code\")\n", + "print(theta)" ] }, { "cell_type": "markdown", - "id": "f048d5c7", + "id": "ba81bc5e", "metadata": { "editable": true }, @@ -2563,7 +2563,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "d8bdd180", + "id": "426cbf5d", "metadata": { "collapsed": false, "editable": true @@ -2622,7 +2622,7 @@ }, { "cell_type": "markdown", - "id": "145ce394", + "id": "fe31ea1d", "metadata": { "editable": true }, @@ -2632,7 +2632,7 @@ }, { "cell_type": "markdown", - "id": "ec0ee110", + "id": "612ae085", "metadata": { "editable": true }, @@ -2643,7 +2643,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "b6e8e3e4", + "id": "8aae6512", "metadata": { "collapsed": false, "editable": true @@ -2708,7 +2708,7 @@ }, { "cell_type": "markdown", - "id": "79d8c0ae", + "id": "1f42017f", "metadata": { "editable": true }, @@ -2719,7 +2719,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "c16ec5ac", + "id": "c298a7cd", "metadata": { "collapsed": false, "editable": true @@ -2760,9 +2760,9 @@ "\n", "# Value for learning rate\n", "eta = 0.01\n", - "# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980\n", - "beta1 = 0.9\n", - "beta2 = 0.999\n", + "# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980\n", + "theta1 = 0.9\n", + "theta2 = 0.999\n", "# Including AdaGrad parameter to avoid possible division by zero\n", "delta = 1e-7\n", "iter = 0\n", @@ -2776,10 +2776,10 @@ " yi = y[random_index:random_index+M]\n", " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n", " # Computing moments first\n", - " first_moment = beta1*first_moment + (1-beta1)*gradients\n", - " second_moment = beta2*second_moment+(1-beta2)*gradients*gradients\n", - " first_term = first_moment/(1.0-beta1**iter)\n", - " second_term = second_moment/(1.0-beta2**iter)\n", + " first_moment = theta1*first_moment + (1-theta1)*gradients\n", + " second_moment = theta2*second_moment+(1-theta2)*gradients*gradients\n", + " first_term = first_moment/(1.0-theta1**iter)\n", + " second_term = second_moment/(1.0-theta2**iter)\n", "\t# Scaling with rho the new and the previous results\n", " update = eta*first_term/(np.sqrt(second_term)+delta)\n", " theta -= update\n", @@ -2789,7 +2789,7 @@ }, { "cell_type": "markdown", - "id": "7c581f8b", + "id": "01dac0f9", "metadata": { "editable": true }, @@ -2801,12 +2801,12 @@ "2. Work on project 1\n", "\n", "\n", - " * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." + "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." ] }, { "cell_type": "markdown", - "id": "d8a82e01", + "id": "603af8bb", "metadata": { "editable": true }, @@ -2834,7 +2834,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "3fa174a9", + "id": "45c8f57c", "metadata": { "collapsed": false, "editable": true @@ -2854,7 +2854,7 @@ }, { "cell_type": "markdown", - "id": "b02de7a6", + "id": "4b43b0d9", "metadata": { "editable": true }, @@ -2870,7 +2870,7 @@ }, { "cell_type": "markdown", - "id": "cd3f85ef", + "id": "0ca981f4", "metadata": { "editable": true }, @@ -2890,7 +2890,7 @@ }, { "cell_type": "markdown", - "id": "1e0198a0", + "id": "2b8965d7", "metadata": { "editable": true }, @@ -2917,7 +2917,7 @@ }, { "cell_type": "markdown", - "id": "04da94ca", + "id": "f9d735f8", "metadata": { "editable": true }, @@ -2930,7 +2930,7 @@ }, { "cell_type": "markdown", - "id": "aa1ad257", + "id": "438722c0", "metadata": { "editable": true }, @@ -2942,7 +2942,7 @@ }, { "cell_type": "markdown", - "id": "6587016d", + "id": "26e667bc", "metadata": { "editable": true }, @@ -2957,7 +2957,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "2e0350b4", + "id": "8d8a76a2", "metadata": { "collapsed": false, "editable": true @@ -2984,7 +2984,7 @@ }, { "cell_type": "markdown", - "id": "8179b037", + "id": "d4213652", "metadata": { "editable": true }, @@ -2998,7 +2998,7 @@ }, { "cell_type": "markdown", - "id": "3ebed257", + "id": "8bac9557", "metadata": { "editable": true }, @@ -3010,7 +3010,7 @@ }, { "cell_type": "markdown", - "id": "51b239a4", + "id": "5848e568", "metadata": { "editable": true }, @@ -3027,7 +3027,7 @@ }, { "cell_type": "markdown", - "id": "295790f3", + "id": "ec41cd20", "metadata": { "editable": true }, @@ -3039,7 +3039,7 @@ }, { "cell_type": "markdown", - "id": "b68338d6", + "id": "25e53ab6", "metadata": { "editable": true }, @@ -3049,7 +3049,7 @@ }, { "cell_type": "markdown", - "id": "c015cb3b", + "id": "eef573a2", "metadata": { "editable": true }, @@ -3061,7 +3061,7 @@ }, { "cell_type": "markdown", - "id": "5e4917fa", + "id": "1f02e0c5", "metadata": { "editable": true }, @@ -3071,7 +3071,7 @@ }, { "cell_type": "markdown", - "id": "150fedf5", + "id": "ceeaec5a", "metadata": { "editable": true }, @@ -3083,7 +3083,7 @@ }, { "cell_type": "markdown", - "id": "e6d782b1", + "id": "d77c3425", "metadata": { "editable": true }, @@ -3094,7 +3094,7 @@ }, { "cell_type": "markdown", - "id": "a4065dd3", + "id": "602222b1", "metadata": { "editable": true }, @@ -3106,7 +3106,7 @@ }, { "cell_type": "markdown", - "id": "06324d79", + "id": "a523d98b", "metadata": { "editable": true }, @@ -3116,7 +3116,7 @@ }, { "cell_type": "markdown", - "id": "9e3e6b35", + "id": "c9705382", "metadata": { "editable": true }, @@ -3128,7 +3128,7 @@ }, { "cell_type": "markdown", - "id": "cd9c3754", + "id": "9aeda995", "metadata": { "editable": true }, @@ -3138,7 +3138,7 @@ }, { "cell_type": "markdown", - "id": "8361829b", + "id": "02edb542", "metadata": { "editable": true }, @@ -3150,7 +3150,7 @@ }, { "cell_type": "markdown", - "id": "8f5f46fd", + "id": "131d33c8", "metadata": { "editable": true }, @@ -3160,7 +3160,7 @@ }, { "cell_type": "markdown", - "id": "cf210a6e", + "id": "1dbf2440", "metadata": { "editable": true }, @@ -3172,7 +3172,7 @@ }, { "cell_type": "markdown", - "id": "d3af09fc", + "id": "5ce1b2a9", "metadata": { "editable": true }, @@ -3182,7 +3182,7 @@ }, { "cell_type": "markdown", - "id": "ec91f8c1", + "id": "102287b0", "metadata": { "editable": true }, @@ -3194,7 +3194,7 @@ }, { "cell_type": "markdown", - "id": "0c9f35e6", + "id": "13e174c8", "metadata": { "editable": true }, @@ -3204,7 +3204,7 @@ }, { "cell_type": "markdown", - "id": "de80fa4b", + "id": "b35bb3c6", "metadata": { "editable": true }, @@ -3216,7 +3216,7 @@ }, { "cell_type": "markdown", - "id": "34c99b80", + "id": "a167affa", "metadata": { "editable": true }, @@ -3226,7 +3226,7 @@ }, { "cell_type": "markdown", - "id": "758ade4d", + "id": "c430eab0", "metadata": { "editable": true }, @@ -3238,7 +3238,7 @@ }, { "cell_type": "markdown", - "id": "ff0f4725", + "id": "5acd5488", "metadata": { "editable": true }, @@ -3248,7 +3248,7 @@ }, { "cell_type": "markdown", - "id": "997b829b", + "id": "28e90177", "metadata": { "editable": true }, @@ -3260,7 +3260,7 @@ }, { "cell_type": "markdown", - "id": "621e7d85", + "id": "8f6c30bc", "metadata": { "editable": true }, @@ -3272,7 +3272,7 @@ }, { "cell_type": "markdown", - "id": "5530e459", + "id": "a7bbebcc", "metadata": { "editable": true }, @@ -3284,7 +3284,7 @@ }, { "cell_type": "markdown", - "id": "fd9bace8", + "id": "98b55049", "metadata": { "editable": true }, @@ -3294,7 +3294,7 @@ }, { "cell_type": "markdown", - "id": "d949be75", + "id": "6970ed00", "metadata": { "editable": true }, @@ -3306,7 +3306,7 @@ }, { "cell_type": "markdown", - "id": "4ad52617", + "id": "5b1a3945", "metadata": { "editable": true }, @@ -3319,7 +3319,7 @@ }, { "cell_type": "markdown", - "id": "41cabc72", + "id": "d64296dd", "metadata": { "editable": true }, @@ -3331,7 +3331,7 @@ }, { "cell_type": "markdown", - "id": "1fd416a6", + "id": "cb226490", "metadata": { "editable": true }, @@ -3345,7 +3345,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "f13cacb4", + "id": "dd41d7f1", "metadata": { "collapsed": false, "editable": true @@ -3442,7 +3442,7 @@ }, { "cell_type": "markdown", - "id": "ff4b28ed", + "id": "eaae6f40", "metadata": { "editable": true }, @@ -3463,7 +3463,7 @@ }, { "cell_type": "markdown", - "id": "03b4dc3c", + "id": "1d8b43a7", "metadata": { "editable": true }, @@ -3475,7 +3475,7 @@ }, { "cell_type": "markdown", - "id": "6cff2148", + "id": "3ae4a36e", "metadata": { "editable": true }, @@ -3485,7 +3485,7 @@ }, { "cell_type": "markdown", - "id": "51d823c4", + "id": "67f213bb", "metadata": { "editable": true }, @@ -3497,7 +3497,7 @@ }, { "cell_type": "markdown", - "id": "5ab270f9", + "id": "bad17614", "metadata": { "editable": true }, @@ -3507,7 +3507,7 @@ }, { "cell_type": "markdown", - "id": "2b2e1492", + "id": "65c9d9e0", "metadata": { "editable": true }, @@ -3519,7 +3519,7 @@ }, { "cell_type": "markdown", - "id": "063be947", + "id": "c1b54c7f", "metadata": { "editable": true }, @@ -3537,7 +3537,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "5e841a10", + "id": "9e7636c4", "metadata": { "collapsed": false, "editable": true @@ -3613,7 +3613,7 @@ }, { "cell_type": "markdown", - "id": "40e54083", + "id": "fb4524b3", "metadata": { "editable": true }, @@ -3627,7 +3627,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "edd7dd42", + "id": "d60aae5a", "metadata": { "collapsed": false, "editable": true @@ -3716,7 +3716,7 @@ }, { "cell_type": "markdown", - "id": "f9cc046b", + "id": "901653e1", "metadata": { "editable": true }, diff --git a/doc/src/week37/week37.do.txt b/doc/src/week37/week37.do.txt index e36c81a3f..0a5334f08 100644 --- a/doc/src/week37/week37.do.txt +++ b/doc/src/week37/week37.do.txt @@ -543,8 +543,8 @@ function, which we want to minimize, can almost always be written as a sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$, !bt \[ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). +C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\theta}). \] !et @@ -555,8 +555,8 @@ This in turn means that the gradient can be computed as a sum over $i$-gradients !bt \[ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). +\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}). \] !et @@ -584,10 +584,10 @@ all data points with a sum over the data points in one the minibatches picked at random in each gradient descent step !bt \[ -\nabla_{\beta} -C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta -c_i(\mathbf{x}_i, \mathbf{\beta}). +\nabla_{\theta} +C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta +c_i(\mathbf{x}_i, \mathbf{\theta}). \] !et @@ -597,8 +597,8 @@ c_i(\mathbf{x}_i, \mathbf{\beta}). Thus a gradient descent step now looks like !bt \[ -\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) +\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \] !et @@ -647,7 +647,7 @@ is zero is valid also for local minima, so this would only tell us that we are close to a local/global minimum. However, we could also evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can -compare the values of the cost function and keep the $\beta$ that +compare the values of the cost function and keep the $\theta$ that gave the lowest value. !split @@ -668,10 +668,10 @@ for a discussion of different scaling functions for the learning rate. As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$. -In this way we can fix the number of epochs, compute $\beta$ and +In this way we can fix the number of epochs, compute $\theta$ and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we -pick the final $\beta$ that gives the lowest value of the cost +pick the final $\theta$ that gives the lowest value of the cost function. !bc pycod @@ -693,7 +693,7 @@ for epoch in range(1,n_epochs+1): for i in range(m): k = np.random.randint(m) #Pick the k-th minibatch at random #Compute the gradient using the data in minibatch Bk - #Compute new suggestion for beta + #Compute new suggestion for theta t = epoch*m+i gamma_j = step_length(t,t0,t1) j += 1 @@ -1005,10 +1005,10 @@ Addresses AdaGrad’s diminishing learning rate issue. Uses a decaying average of squared gradients (instead of a cumulative sum): !bt \[ -v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2, +v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2, \] !et -with $\beta_2$ typically $0.9$ (or $0.99$). +with $\theta_2$ typically $0.9$ (or $0.99$). o Update: $w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t)$. o Recent gradients have more weight, so $v_t$ adapts to the current landscape. o Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero. @@ -1039,13 +1039,13 @@ for RMS prop is given by !bt \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\ -\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\ +\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\ \boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber \end{align} !et -where $\beta$ controls the averaging time of the second moment and is -typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate +where $\theta$ controls the averaging time of the second moment and is +typically taken to be about $\theta=0.9$, $\eta_t$ is a learning rate typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It @@ -1080,15 +1080,15 @@ again understood to be element-wise operations below) !bt \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\ -\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ -\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ -\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ -\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\ +\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\ +\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\ +\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\ \boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \bm{\mathbf{m}}_t \over \sqrt{\bm{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \end{align} !et -where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and +where $\theta_1$ and $\theta_2$ set the memory lifetime of the first and second moment and are typically taken to be $0.9$ and $0.99$ respectively, and $\eta$ and $\epsilon$ are identical to RMSprop. @@ -1165,8 +1165,8 @@ import autograd.numpy as np import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1221,8 +1221,8 @@ import autograd.numpy as np import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1440,8 +1440,8 @@ import numpy as np import autograd.numpy as np from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1449,24 +1449,24 @@ y = 4+3*x+5*x*x X = np.c_[np.ones((n,1)), x, x*x] XT_X = X.T @ X -beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) +theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) print("Own inversion") -print(beta_linreg) +print(theta_linreg) # Hessian matrix H = (2.0/n)* XT_X -# Note that here the Hessian does not depend on the parameters beta +# Note that here the Hessian does not depend on the parameters theta invH = np.linalg.pinv(H) -beta = np.random.randn(3,1) +theta = np.random.randn(3,1) Niterations = 5 # define the gradient training_gradient = grad(CostOLS) for iter in range(Niterations): - gradients = training_gradient(beta) - beta -= invH @ gradients + gradients = training_gradient(theta) + theta -= invH @ gradients print(iter,gradients[0],gradients[1]) -print("beta from own Newton code") -print(beta) +print("theta from own Newton code") +print(theta) !ec @@ -1626,9 +1626,9 @@ theta = np.random.randn(3,1) # Value for learning rate eta = 0.01 -# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980 -beta1 = 0.9 -beta2 = 0.999 +# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980 +theta1 = 0.9 +theta2 = 0.999 # Including AdaGrad parameter to avoid possible division by zero delta = 1e-7 iter = 0 @@ -1642,10 +1642,10 @@ for epoch in range(n_epochs): yi = y[random_index:random_index+M] gradients = (1.0/M)*training_gradient(yi, xi, theta) # Computing moments first - first_moment = beta1*first_moment + (1-beta1)*gradients - second_moment = beta2*second_moment+(1-beta2)*gradients*gradients - first_term = first_moment/(1.0-beta1**iter) - second_term = second_moment/(1.0-beta2**iter) + first_moment = theta1*first_moment + (1-theta1)*gradients + second_moment = theta2*second_moment+(1-theta2)*gradients*gradients + first_term = first_moment/(1.0-theta1**iter) + second_term = second_moment/(1.0-theta2**iter) # Scaling with rho the new and the previous results update = eta*first_term/(np.sqrt(second_term)+delta) theta -= update @@ -1662,7 +1662,7 @@ print(theta) o Exercise set for week 37 and reminder on scaling (from lab sessions of week 35) o Work on project 1 # * "Video of exercise sessions week 37":"https://youtu.be/bK4AEcTu-oM" - * For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended. +For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended. !eblock