lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

changed variable names

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-07 19:55:00 +02:00
parent 831312ce60
commit f27b2c9260
17 changed files with 454 additions and 461 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week37/html/._week37-bs022.html
+2 -2
View File
@@ -339,8 +339,8 @@ function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
doc/pub/week37/html/._week37-bs023.html
+2 -2
View File
@@ -334,8 +334,8 @@ MathJax.Hub.Config({
computed as a sum over \( i \)-gradients
</p>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
<p>Stochasticity/randomness is introduced by only taking the
doc/pub/week37/html/._week37-bs024.html
+4 -4
View File
@@ -344,10 +344,10 @@ all data points with a sum over the data points in one the minibatches
picked at random in each gradient descent step
</p>
$$
\nabla_{\beta}
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
\nabla_{\theta}
C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta
c_i(\mathbf{x}_i, \mathbf{\theta}).
$$
doc/pub/week37/html/._week37-bs025.html
+2 -2
View File
@@ -332,8 +332,8 @@ MathJax.Hub.Config({
<p>Thus a gradient descent step now looks like </p>
$$
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta})
$$
<p>where \( k \) is picked at random with equal
doc/pub/week37/html/._week37-bs027.html
+1 -1
View File
@@ -338,7 +338,7 @@ is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the \( \beta \) that
compare the values of the cost function and keep the \( \theta \) that
gave the lowest value.
</p>
doc/pub/week37/html/._week37-bs029.html
+3 -3
View File
@@ -332,10 +332,10 @@ MathJax.Hub.Config({
<p>As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in <em>time</em> \( t \).</p>
<p>In this way we can fix the number of epochs, compute \( \beta \) and
<p>In this way we can fix the number of epochs, compute \( \theta \) and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final \( \beta \) that gives the lowest value of the cost
pick the final \( \theta \) that gives the lowest value of the cost
function.
</p>
@@ -364,7 +364,7 @@ j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(m):
k <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randint(m) <span style="color: #408080; font-style: italic">#Pick the k-th minibatch at random</span>
<span style="color: #408080; font-style: italic">#Compute the gradient using the data in minibatch Bk</span>
<span style="color: #408080; font-style: italic">#Compute new suggestion for beta</span>
<span style="color: #408080; font-style: italic">#Compute new suggestion for theta</span>
t <span style="color: #666666">=</span> epoch<span style="color: #666666">*</span>m<span style="color: #666666">+</span>i
gamma_j <span style="color: #666666">=</span> step_length(t,t0,t1)
j <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
doc/pub/week37/html/._week37-bs041.html
+2 -2
View File
@@ -333,10 +333,10 @@ MathJax.Hub.Config({
Uses a decaying average of squared gradients (instead of a cumulative sum):
</p>
$$
v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2,
v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2,
$$
<p>with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<p>with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<ol>
<li> Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).</li>
<li> Recent gradients have more weight, so \( v_t \) adapts to the current landscape.</li>
doc/pub/week37/html/._week37-bs043.html
+3 -3
View File
@@ -340,13 +340,13 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\tag{3}\\
\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\
\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
$$
<p>where \( \beta \) controls the averaging time of the second moment and is
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
<p>where \( \theta \) controls the averaging time of the second moment and is
typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
small regularization constant to prevent divergences. Multiplication
and division by vectors is understood as an element-wise operation. It
doc/pub/week37/html/._week37-bs044.html
+5 -5
View File
@@ -355,16 +355,16 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\tag{4}\\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\tag{5}
\end{align}
$$
<p>where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
<p>where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
</p>
doc/pub/week37/html/._week37-bs047.html
+2 -2
View File
@@ -356,8 +356,8 @@ first example shows results with ordinary leats squares.
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(beta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> beta)<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(theta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> theta)<span style="color: #666666">**2</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(n,<span style="color: #666666">1</span>)
doc/pub/week37/html/._week37-bs048.html
+2 -2
View File
@@ -343,8 +343,8 @@ MathJax.Hub.Config({
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(beta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> beta)<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(theta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> theta)<span style="color: #666666">**2</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(n,<span style="color: #666666">1</span>)
doc/pub/week37/html/week37-reveal.html
+47 -50
View File
@@ -856,8 +856,8 @@ sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
<p>&nbsp;<br>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
<p>&nbsp;<br>
</section>
@@ -870,8 +870,8 @@ computed as a sum over \( i \)-gradients
</p>
<p>&nbsp;<br>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
<p>&nbsp;<br>
@@ -901,10 +901,10 @@ picked at random in each gradient descent step
</p>
<p>&nbsp;<br>
$$
\nabla_{\beta}
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
\nabla_{\theta}
C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta
c_i(\mathbf{x}_i, \mathbf{\theta}).
$$
<p>&nbsp;<br>
</section>
@@ -915,8 +915,8 @@ $$
<p>Thus a gradient descent step now looks like </p>
<p>&nbsp;<br>
$$
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta})
$$
<p>&nbsp;<br>
@@ -988,7 +988,7 @@ is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the \( \beta \) that
compare the values of the cost function and keep the \( \theta \) that
gave the lowest value.
</p>
</section>
@@ -1015,10 +1015,10 @@ for a discussion of different scaling functions for the learning rate.
$$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$
<p>&nbsp;<br> goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in <em>time</em> \( t \).</p>
<p>In this way we can fix the number of epochs, compute \( \beta \) and
<p>In this way we can fix the number of epochs, compute \( \theta \) and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final \( \beta \) that gives the lowest value of the cost
pick the final \( \theta \) that gives the lowest value of the cost
function.
</p>
@@ -1047,7 +1047,7 @@ j = <span style="color: #B452CD">0</span>
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(m):
k = np.random.randint(m) <span style="color: #228B22">#Pick the k-th minibatch at random</span>
<span style="color: #228B22">#Compute the gradient using the data in minibatch Bk</span>
<span style="color: #228B22">#Compute new suggestion for beta</span>
<span style="color: #228B22">#Compute new suggestion for theta</span>
t = epoch*m+i
gamma_j = step_length(t,t0,t1)
j += <span style="color: #B452CD">1</span>
@@ -1439,11 +1439,11 @@ Uses a decaying average of squared gradients (instead of a cumulative sum):
</p>
<p>&nbsp;<br>
$$
v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2,
v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2,
$$
<p>&nbsp;<br>
<p>with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<p>with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<ol>
<p><li> Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).</li>
<p><li> Recent gradients have more weight, so \( v_t \) adapts to the current landscape.</li>
@@ -1482,14 +1482,14 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\tag{3}\\
\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\
\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
$$
<p>&nbsp;<br>
<p>where \( \beta \) controls the averaging time of the second moment and is
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
<p>where \( \theta \) controls the averaging time of the second moment and is
typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
small regularization constant to prevent divergences. Multiplication
and division by vectors is understood as an element-wise operation. It
@@ -1529,17 +1529,17 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\tag{4}\\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\tag{5}
\end{align}
$$
<p>&nbsp;<br>
<p>where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
<p>where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
</p>
@@ -1628,8 +1628,8 @@ first example shows results with ordinary leats squares.
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(beta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ beta)**<span style="color: #B452CD">2</span>)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(theta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ theta)**<span style="color: #B452CD">2</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
@@ -1702,8 +1702,8 @@ plt.show()
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(beta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ beta)**<span style="color: #B452CD">2</span>)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(theta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ theta)**<span style="color: #B452CD">2</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
@@ -1977,8 +1977,8 @@ delta_momentum = <span style="color: #B452CD">0.3</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(beta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ beta)**<span style="color: #B452CD">2</span>)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(theta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ theta)**<span style="color: #B452CD">2</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
@@ -1986,24 +1986,24 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
X = np.c_[np.ones((n,<span style="color: #B452CD">1</span>)), x, x*x]
XT_X = X.T @ X
beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Own inversion&quot;</span>)
<span style="color: #658b00">print</span>(beta_linreg)
<span style="color: #658b00">print</span>(theta_linreg)
<span style="color: #228B22"># Hessian matrix</span>
H = (<span style="color: #B452CD">2.0</span>/n)* XT_X
<span style="color: #228B22"># Note that here the Hessian does not depend on the parameters beta</span>
<span style="color: #228B22"># Note that here the Hessian does not depend on the parameters theta</span>
invH = np.linalg.pinv(H)
beta = np.random.randn(<span style="color: #B452CD">3</span>,<span style="color: #B452CD">1</span>)
theta = np.random.randn(<span style="color: #B452CD">3</span>,<span style="color: #B452CD">1</span>)
Niterations = <span style="color: #B452CD">5</span>
<span style="color: #228B22"># define the gradient</span>
training_gradient = grad(CostOLS)
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = training_gradient(beta)
beta -= invH @ gradients
gradients = training_gradient(theta)
theta -= invH @ gradients
<span style="color: #658b00">print</span>(<span style="color: #658b00">iter</span>,gradients[<span style="color: #B452CD">0</span>],gradients[<span style="color: #B452CD">1</span>])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;beta from own Newton code&quot;</span>)
<span style="color: #658b00">print</span>(beta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;theta from own Newton code&quot;</span>)
<span style="color: #658b00">print</span>(theta)
</pre>
</div>
</div>
@@ -2218,9 +2218,9 @@ theta = np.random.randn(<span style="color: #B452CD">3</span>,<span style="color
<span style="color: #228B22"># Value for learning rate</span>
eta = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980</span>
beta1 = <span style="color: #B452CD">0.9</span>
beta2 = <span style="color: #B452CD">0.999</span>
<span style="color: #228B22"># Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980</span>
theta1 = <span style="color: #B452CD">0.9</span>
theta2 = <span style="color: #B452CD">0.999</span>
<span style="color: #228B22"># Including AdaGrad parameter to avoid possible division by zero</span>
delta = <span style="color: #B452CD">1e-7</span>
<span style="color: #658b00">iter</span> = <span style="color: #B452CD">0</span>
@@ -2234,10 +2234,10 @@ delta = <span style="color: #B452CD">1e-7</span>
yi = y[random_index:random_index+M]
gradients = (<span style="color: #B452CD">1.0</span>/M)*training_gradient(yi, xi, theta)
<span style="color: #228B22"># Computing moments first</span>
first_moment = beta1*first_moment + (<span style="color: #B452CD">1</span>-beta1)*gradients
second_moment = beta2*second_moment+(<span style="color: #B452CD">1</span>-beta2)*gradients*gradients
first_term = first_moment/(<span style="color: #B452CD">1.0</span>-beta1**<span style="color: #658b00">iter</span>)
second_term = second_moment/(<span style="color: #B452CD">1.0</span>-beta2**<span style="color: #658b00">iter</span>)
first_moment = theta1*first_moment + (<span style="color: #B452CD">1</span>-theta1)*gradients
second_moment = theta2*second_moment+(<span style="color: #B452CD">1</span>-theta2)*gradients*gradients
first_term = first_moment/(<span style="color: #B452CD">1.0</span>-theta1**<span style="color: #658b00">iter</span>)
second_term = second_moment/(<span style="color: #B452CD">1.0</span>-theta2**<span style="color: #658b00">iter</span>)
<span style="color: #228B22"># Scaling with rho the new and the previous results</span>
update = eta*first_term/(np.sqrt(second_term)+delta)
theta -= update
@@ -2269,12 +2269,9 @@ delta = <span style="color: #B452CD">1e-7</span>
<p><li> Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)</li>
<p><li> Work on project 1
<!-- * <a href="https://youtu.be/bK4AEcTu-oM" target="_blank">Video of exercise sessions week 37</a> --></li>
<ul>
<p><li> For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</li>
</ul>
<p>
</ol>
<p>
<p>For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</p>
</div>
</section>
doc/pub/week37/html/week37-solarized.html
+46 -48
View File
@@ -909,8 +909,8 @@ function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
@@ -921,8 +921,8 @@ $$
computed as a sum over \( i \)-gradients
</p>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
<p>Stochasticity/randomness is introduced by only taking the
@@ -949,10 +949,10 @@ all data points with a sum over the data points in one the minibatches
picked at random in each gradient descent step
</p>
$$
\nabla_{\beta}
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
\nabla_{\theta}
C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta
c_i(\mathbf{x}_i, \mathbf{\theta}).
$$
@@ -961,8 +961,8 @@ $$
<p>Thus a gradient descent step now looks like </p>
$$
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta})
$$
<p>where \( k \) is picked at random with equal
@@ -1031,7 +1031,7 @@ is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the \( \beta \) that
compare the values of the cost function and keep the \( \theta \) that
gave the lowest value.
</p>
@@ -1054,10 +1054,10 @@ for a discussion of different scaling functions for the learning rate.
<p>As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in <em>time</em> \( t \).</p>
<p>In this way we can fix the number of epochs, compute \( \beta \) and
<p>In this way we can fix the number of epochs, compute \( \theta \) and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final \( \beta \) that gives the lowest value of the cost
pick the final \( \theta \) that gives the lowest value of the cost
function.
</p>
@@ -1086,7 +1086,7 @@ j = <span style="color: #B452CD">0</span>
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(m):
k = np.random.randint(m) <span style="color: #228B22">#Pick the k-th minibatch at random</span>
<span style="color: #228B22">#Compute the gradient using the data in minibatch Bk</span>
<span style="color: #228B22">#Compute new suggestion for beta</span>
<span style="color: #228B22">#Compute new suggestion for theta</span>
t = epoch*m+i
gamma_j = step_length(t,t0,t1)
j += <span style="color: #B452CD">1</span>
@@ -1441,10 +1441,10 @@ $$
Uses a decaying average of squared gradients (instead of a cumulative sum):
</p>
$$
v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2,
v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2,
$$
<p>with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<p>with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<ol>
<li> Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).</li>
<li> Recent gradients have more weight, so \( v_t \) adapts to the current landscape.</li>
@@ -1478,13 +1478,13 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\label{_auto3}\\
\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\
\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
$$
<p>where \( \beta \) controls the averaging time of the second moment and is
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
<p>where \( \theta \) controls the averaging time of the second moment and is
typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
small regularization constant to prevent divergences. Multiplication
and division by vectors is understood as an element-wise operation. It
@@ -1522,16 +1522,16 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\label{_auto4}\\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\label{_auto5}
\end{align}
$$
<p>where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
<p>where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
</p>
@@ -1615,8 +1615,8 @@ first example shows results with ordinary leats squares.
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(beta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ beta)**<span style="color: #B452CD">2</span>)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(theta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ theta)**<span style="color: #B452CD">2</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
@@ -1689,8 +1689,8 @@ plt.show()
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(beta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ beta)**<span style="color: #B452CD">2</span>)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(theta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ theta)**<span style="color: #B452CD">2</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
@@ -1964,8 +1964,8 @@ delta_momentum = <span style="color: #B452CD">0.3</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(beta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ beta)**<span style="color: #B452CD">2</span>)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">CostOLS</span>(theta):
<span style="color: #8B008B; font-weight: bold">return</span> (<span style="color: #B452CD">1.0</span>/n)*np.sum((y-X @ theta)**<span style="color: #B452CD">2</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
@@ -1973,24 +1973,24 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
X = np.c_[np.ones((n,<span style="color: #B452CD">1</span>)), x, x*x]
XT_X = X.T @ X
beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Own inversion&quot;</span>)
<span style="color: #658b00">print</span>(beta_linreg)
<span style="color: #658b00">print</span>(theta_linreg)
<span style="color: #228B22"># Hessian matrix</span>
H = (<span style="color: #B452CD">2.0</span>/n)* XT_X
<span style="color: #228B22"># Note that here the Hessian does not depend on the parameters beta</span>
<span style="color: #228B22"># Note that here the Hessian does not depend on the parameters theta</span>
invH = np.linalg.pinv(H)
beta = np.random.randn(<span style="color: #B452CD">3</span>,<span style="color: #B452CD">1</span>)
theta = np.random.randn(<span style="color: #B452CD">3</span>,<span style="color: #B452CD">1</span>)
Niterations = <span style="color: #B452CD">5</span>
<span style="color: #228B22"># define the gradient</span>
training_gradient = grad(CostOLS)
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = training_gradient(beta)
beta -= invH @ gradients
gradients = training_gradient(theta)
theta -= invH @ gradients
<span style="color: #658b00">print</span>(<span style="color: #658b00">iter</span>,gradients[<span style="color: #B452CD">0</span>],gradients[<span style="color: #B452CD">1</span>])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;beta from own Newton code&quot;</span>)
<span style="color: #658b00">print</span>(beta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;theta from own Newton code&quot;</span>)
<span style="color: #658b00">print</span>(theta)
</pre>
</div>
</div>
@@ -2204,9 +2204,9 @@ theta = np.random.randn(<span style="color: #B452CD">3</span>,<span style="color
<span style="color: #228B22"># Value for learning rate</span>
eta = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980</span>
beta1 = <span style="color: #B452CD">0.9</span>
beta2 = <span style="color: #B452CD">0.999</span>
<span style="color: #228B22"># Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980</span>
theta1 = <span style="color: #B452CD">0.9</span>
theta2 = <span style="color: #B452CD">0.999</span>
<span style="color: #228B22"># Including AdaGrad parameter to avoid possible division by zero</span>
delta = <span style="color: #B452CD">1e-7</span>
<span style="color: #658b00">iter</span> = <span style="color: #B452CD">0</span>
@@ -2220,10 +2220,10 @@ delta = <span style="color: #B452CD">1e-7</span>
yi = y[random_index:random_index+M]
gradients = (<span style="color: #B452CD">1.0</span>/M)*training_gradient(yi, xi, theta)
<span style="color: #228B22"># Computing moments first</span>
first_moment = beta1*first_moment + (<span style="color: #B452CD">1</span>-beta1)*gradients
second_moment = beta2*second_moment+(<span style="color: #B452CD">1</span>-beta2)*gradients*gradients
first_term = first_moment/(<span style="color: #B452CD">1.0</span>-beta1**<span style="color: #658b00">iter</span>)
second_term = second_moment/(<span style="color: #B452CD">1.0</span>-beta2**<span style="color: #658b00">iter</span>)
first_moment = theta1*first_moment + (<span style="color: #B452CD">1</span>-theta1)*gradients
second_moment = theta2*second_moment+(<span style="color: #B452CD">1</span>-theta2)*gradients*gradients
first_term = first_moment/(<span style="color: #B452CD">1.0</span>-theta1**<span style="color: #658b00">iter</span>)
second_term = second_moment/(<span style="color: #B452CD">1.0</span>-theta2**<span style="color: #658b00">iter</span>)
<span style="color: #228B22"># Scaling with rho the new and the previous results</span>
update = eta*first_term/(np.sqrt(second_term)+delta)
theta -= update
@@ -2255,10 +2255,8 @@ delta = <span style="color: #B452CD">1e-7</span>
<li> Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)</li>
<li> Work on project 1
<!-- * <a href="https://youtu.be/bK4AEcTu-oM" target="_blank">Video of exercise sessions week 37</a> --></li>
<ul>
<li> For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</li>
</ul>
</ol>
<p>For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</p>
</div>
doc/pub/week37/html/week37.html
+46 -48
View File
@@ -986,8 +986,8 @@ function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
@@ -998,8 +998,8 @@ $$
computed as a sum over \( i \)-gradients
</p>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}).
$$
<p>Stochasticity/randomness is introduced by only taking the
@@ -1026,10 +1026,10 @@ all data points with a sum over the data points in one the minibatches
picked at random in each gradient descent step
</p>
$$
\nabla_{\beta}
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
\nabla_{\theta}
C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta
c_i(\mathbf{x}_i, \mathbf{\theta}).
$$
@@ -1038,8 +1038,8 @@ $$
<p>Thus a gradient descent step now looks like </p>
$$
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta})
$$
<p>where \( k \) is picked at random with equal
@@ -1108,7 +1108,7 @@ is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the \( \beta \) that
compare the values of the cost function and keep the \( \theta \) that
gave the lowest value.
</p>
@@ -1131,10 +1131,10 @@ for a discussion of different scaling functions for the learning rate.
<p>As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in <em>time</em> \( t \).</p>
<p>In this way we can fix the number of epochs, compute \( \beta \) and
<p>In this way we can fix the number of epochs, compute \( \theta \) and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final \( \beta \) that gives the lowest value of the cost
pick the final \( \theta \) that gives the lowest value of the cost
function.
</p>
@@ -1163,7 +1163,7 @@ j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(m):
k <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randint(m) <span style="color: #408080; font-style: italic">#Pick the k-th minibatch at random</span>
<span style="color: #408080; font-style: italic">#Compute the gradient using the data in minibatch Bk</span>
<span style="color: #408080; font-style: italic">#Compute new suggestion for beta</span>
<span style="color: #408080; font-style: italic">#Compute new suggestion for theta</span>
t <span style="color: #666666">=</span> epoch<span style="color: #666666">*</span>m<span style="color: #666666">+</span>i
gamma_j <span style="color: #666666">=</span> step_length(t,t0,t1)
j <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
@@ -1518,10 +1518,10 @@ $$
Uses a decaying average of squared gradients (instead of a cumulative sum):
</p>
$$
v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2,
v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2,
$$
<p>with \( \beta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<p>with \( \theta_2 \) typically \( 0.9 \) (or \( 0.99 \)).</p>
<ol>
<li> Update: \( w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t) \).</li>
<li> Recent gradients have more weight, so \( v_t \) adapts to the current landscape.</li>
@@ -1555,13 +1555,13 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\label{_auto3}\\
\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\
\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
$$
<p>where \( \beta \) controls the averaging time of the second moment and is
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
<p>where \( \theta \) controls the averaging time of the second moment and is
typically taken to be about \( \theta=0.9 \), \( \eta_t \) is a learning rate
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
small regularization constant to prevent divergences. Multiplication
and division by vectors is understood as an element-wise operation. It
@@ -1599,16 +1599,16 @@ $$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\label{_auto4}\\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\label{_auto5}
\end{align}
$$
<p>where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
<p>where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
</p>
@@ -1692,8 +1692,8 @@ first example shows results with ordinary leats squares.
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(beta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> beta)<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(theta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> theta)<span style="color: #666666">**2</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(n,<span style="color: #666666">1</span>)
@@ -1766,8 +1766,8 @@ plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(beta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> beta)<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(theta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> theta)<span style="color: #666666">**2</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(n,<span style="color: #666666">1</span>)
@@ -2041,8 +2041,8 @@ delta_momentum <span style="color: #666666">=</span> <span style="color: #666666
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(beta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> beta)<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(theta):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0/</span>n)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>sum((y<span style="color: #666666">-</span>X <span style="color: #666666">@</span> theta)<span style="color: #666666">**2</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(n,<span style="color: #666666">1</span>)
@@ -2050,24 +2050,24 @@ y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((n,<span style="color: #666666">1</span>)), x, x<span style="color: #666666">*</span>x]
XT_X <span style="color: #666666">=</span> X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(XT_X) <span style="color: #666666">@</span> (X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y)
theta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(XT_X) <span style="color: #666666">@</span> (X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Own inversion&quot;</span>)
<span style="color: #008000">print</span>(beta_linreg)
<span style="color: #008000">print</span>(theta_linreg)
<span style="color: #408080; font-style: italic"># Hessian matrix</span>
H <span style="color: #666666">=</span> (<span style="color: #666666">2.0/</span>n)<span style="color: #666666">*</span> XT_X
<span style="color: #408080; font-style: italic"># Note that here the Hessian does not depend on the parameters beta</span>
<span style="color: #408080; font-style: italic"># Note that here the Hessian does not depend on the parameters theta</span>
invH <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(H)
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">3</span>,<span style="color: #666666">1</span>)
theta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">3</span>,<span style="color: #666666">1</span>)
Niterations <span style="color: #666666">=</span> <span style="color: #666666">5</span>
<span style="color: #408080; font-style: italic"># define the gradient</span>
training_gradient <span style="color: #666666">=</span> grad(CostOLS)
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
gradients <span style="color: #666666">=</span> training_gradient(beta)
beta <span style="color: #666666">-=</span> invH <span style="color: #666666">@</span> gradients
gradients <span style="color: #666666">=</span> training_gradient(theta)
theta <span style="color: #666666">-=</span> invH <span style="color: #666666">@</span> gradients
<span style="color: #008000">print</span>(<span style="color: #008000">iter</span>,gradients[<span style="color: #666666">0</span>],gradients[<span style="color: #666666">1</span>])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;beta from own Newton code&quot;</span>)
<span style="color: #008000">print</span>(beta)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;theta from own Newton code&quot;</span>)
<span style="color: #008000">print</span>(theta)
</pre>
</div>
</div>
@@ -2281,9 +2281,9 @@ theta <span style="color: #666666">=</span> np<span style="color: #666666">.</sp
<span style="color: #408080; font-style: italic"># Value for learning rate</span>
eta <span style="color: #666666">=</span> <span style="color: #666666">0.01</span>
<span style="color: #408080; font-style: italic"># Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980</span>
beta1 <span style="color: #666666">=</span> <span style="color: #666666">0.9</span>
beta2 <span style="color: #666666">=</span> <span style="color: #666666">0.999</span>
<span style="color: #408080; font-style: italic"># Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980</span>
theta1 <span style="color: #666666">=</span> <span style="color: #666666">0.9</span>
theta2 <span style="color: #666666">=</span> <span style="color: #666666">0.999</span>
<span style="color: #408080; font-style: italic"># Including AdaGrad parameter to avoid possible division by zero</span>
delta <span style="color: #666666">=</span> <span style="color: #666666">1e-7</span>
<span style="color: #008000">iter</span> <span style="color: #666666">=</span> <span style="color: #666666">0</span>
@@ -2297,10 +2297,10 @@ delta <span style="color: #666666">=</span> <span style="color: #666666">1e-7</
yi <span style="color: #666666">=</span> y[random_index:random_index<span style="color: #666666">+</span>M]
gradients <span style="color: #666666">=</span> (<span style="color: #666666">1.0/</span>M)<span style="color: #666666">*</span>training_gradient(yi, xi, theta)
<span style="color: #408080; font-style: italic"># Computing moments first</span>
first_moment <span style="color: #666666">=</span> beta1<span style="color: #666666">*</span>first_moment <span style="color: #666666">+</span> (<span style="color: #666666">1-</span>beta1)<span style="color: #666666">*</span>gradients
second_moment <span style="color: #666666">=</span> beta2<span style="color: #666666">*</span>second_moment<span style="color: #666666">+</span>(<span style="color: #666666">1-</span>beta2)<span style="color: #666666">*</span>gradients<span style="color: #666666">*</span>gradients
first_term <span style="color: #666666">=</span> first_moment<span style="color: #666666">/</span>(<span style="color: #666666">1.0-</span>beta1<span style="color: #666666">**</span><span style="color: #008000">iter</span>)
second_term <span style="color: #666666">=</span> second_moment<span style="color: #666666">/</span>(<span style="color: #666666">1.0-</span>beta2<span style="color: #666666">**</span><span style="color: #008000">iter</span>)
first_moment <span style="color: #666666">=</span> theta1<span style="color: #666666">*</span>first_moment <span style="color: #666666">+</span> (<span style="color: #666666">1-</span>theta1)<span style="color: #666666">*</span>gradients
second_moment <span style="color: #666666">=</span> theta2<span style="color: #666666">*</span>second_moment<span style="color: #666666">+</span>(<span style="color: #666666">1-</span>theta2)<span style="color: #666666">*</span>gradients<span style="color: #666666">*</span>gradients
first_term <span style="color: #666666">=</span> first_moment<span style="color: #666666">/</span>(<span style="color: #666666">1.0-</span>theta1<span style="color: #666666">**</span><span style="color: #008000">iter</span>)
second_term <span style="color: #666666">=</span> second_moment<span style="color: #666666">/</span>(<span style="color: #666666">1.0-</span>theta2<span style="color: #666666">**</span><span style="color: #008000">iter</span>)
<span style="color: #408080; font-style: italic"># Scaling with rho the new and the previous results</span>
update <span style="color: #666666">=</span> eta<span style="color: #666666">*</span>first_term<span style="color: #666666">/</span>(np<span style="color: #666666">.</span>sqrt(second_term)<span style="color: #666666">+</span>delta)
theta <span style="color: #666666">-=</span> update
@@ -2332,10 +2332,8 @@ delta <span style="color: #666666">=</span> <span style="color: #666666">1e-7</
<li> Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)</li>
<li> Work on project 1
<!-- * <a href="https://youtu.be/bK4AEcTu-oM" target="_blank">Video of exercise sessions week 37</a> --></li>
<ul>
<li> For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</li>
</ul>
</ol>
<p>For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</p>
</div>
doc/pub/week37/ipynb/ipynb-week37-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week37/ipynb/week37.ipynb
+241 -241
View File
File diff suppressed because it is too large Load Diff
doc/src/week37/week37.do.txt
+46 -46
View File
@@ -543,8 +543,8 @@ function, which we want to minimize, can almost always be written as a
sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$,
!bt
\[
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\theta}).
\]
!et
@@ -555,8 +555,8 @@ This in turn means that the gradient can be
computed as a sum over $i$-gradients
!bt
\[
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}).
\]
!et
@@ -584,10 +584,10 @@ all data points with a sum over the data points in one the minibatches
picked at random in each gradient descent step
!bt
\[
\nabla_{\beta}
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
\nabla_{\theta}
C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta
c_i(\mathbf{x}_i, \mathbf{\theta}).
\]
!et
@@ -597,8 +597,8 @@ c_i(\mathbf{x}_i, \mathbf{\beta}).
Thus a gradient descent step now looks like
!bt
\[
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta})
\]
!et
@@ -647,7 +647,7 @@ is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the $\beta$ that
compare the values of the cost function and keep the $\theta$ that
gave the lowest value.
!split
@@ -668,10 +668,10 @@ for a discussion of different scaling functions for the learning rate.
As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.
In this way we can fix the number of epochs, compute $\beta$ and
In this way we can fix the number of epochs, compute $\theta$ and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final $\beta$ that gives the lowest value of the cost
pick the final $\theta$ that gives the lowest value of the cost
function.
!bc pycod
@@ -693,7 +693,7 @@ for epoch in range(1,n_epochs+1):
for i in range(m):
k = np.random.randint(m) #Pick the k-th minibatch at random
#Compute the gradient using the data in minibatch Bk
#Compute new suggestion for beta
#Compute new suggestion for theta
t = epoch*m+i
gamma_j = step_length(t,t0,t1)
j += 1
@@ -1005,10 +1005,10 @@ Addresses AdaGrad’s diminishing learning rate issue.
Uses a decaying average of squared gradients (instead of a cumulative sum):
!bt
\[
v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2,
v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2,
\]
!et
with $\beta_2$ typically $0.9$ (or $0.99$).
with $\theta_2$ typically $0.9$ (or $0.99$).
o Update: $w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t)$.
o Recent gradients have more weight, so $v_t$ adapts to the current landscape.
o Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
@@ -1039,13 +1039,13 @@ for RMS prop is given by
!bt
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\
\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
!et
where $\beta$ controls the averaging time of the second moment and is
typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate
where $\theta$ controls the averaging time of the second moment and is
typically taken to be about $\theta=0.9$, $\eta_t$ is a learning rate
typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a
small regularization constant to prevent divergences. Multiplication
and division by vectors is understood as an element-wise operation. It
@@ -1080,15 +1080,15 @@ again understood to be element-wise operations below)
!bt
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \bm{\mathbf{m}}_t \over \sqrt{\bm{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\end{align}
!et
where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and
where $\theta_1$ and $\theta_2$ set the memory lifetime of the first and
second moment and are typically taken to be $0.9$ and $0.99$
respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
@@ -1165,8 +1165,8 @@ import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
def CostOLS(beta):
return (1.0/n)*np.sum((y-X @ beta)**2)
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
@@ -1221,8 +1221,8 @@ import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
def CostOLS(beta):
return (1.0/n)*np.sum((y-X @ beta)**2)
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
@@ -1440,8 +1440,8 @@ import numpy as np
import autograd.numpy as np
from autograd import grad
def CostOLS(beta):
return (1.0/n)*np.sum((y-X @ beta)**2)
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
@@ -1449,24 +1449,24 @@ y = 4+3*x+5*x*x
X = np.c_[np.ones((n,1)), x, x*x]
XT_X = X.T @ X
beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print("Own inversion")
print(beta_linreg)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
# Note that here the Hessian does not depend on the parameters beta
# Note that here the Hessian does not depend on the parameters theta
invH = np.linalg.pinv(H)
beta = np.random.randn(3,1)
theta = np.random.randn(3,1)
Niterations = 5
# define the gradient
training_gradient = grad(CostOLS)
for iter in range(Niterations):
gradients = training_gradient(beta)
beta -= invH @ gradients
gradients = training_gradient(theta)
theta -= invH @ gradients
print(iter,gradients[0],gradients[1])
print("beta from own Newton code")
print(beta)
print("theta from own Newton code")
print(theta)
!ec
@@ -1626,9 +1626,9 @@ theta = np.random.randn(3,1)
# Value for learning rate
eta = 0.01
# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980
beta1 = 0.9
beta2 = 0.999
# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980
theta1 = 0.9
theta2 = 0.999
# Including AdaGrad parameter to avoid possible division by zero
delta = 1e-7
iter = 0
@@ -1642,10 +1642,10 @@ for epoch in range(n_epochs):
yi = y[random_index:random_index+M]
gradients = (1.0/M)*training_gradient(yi, xi, theta)
# Computing moments first
first_moment = beta1*first_moment + (1-beta1)*gradients
second_moment = beta2*second_moment+(1-beta2)*gradients*gradients
first_term = first_moment/(1.0-beta1**iter)
second_term = second_moment/(1.0-beta2**iter)
first_moment = theta1*first_moment + (1-theta1)*gradients
second_moment = theta2*second_moment+(1-theta2)*gradients*gradients
first_term = first_moment/(1.0-theta1**iter)
second_term = second_moment/(1.0-theta2**iter)
# Scaling with rho the new and the previous results
update = eta*first_term/(np.sqrt(second_term)+delta)
theta -= update
@@ -1662,7 +1662,7 @@ print(theta)
o Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)
o Work on project 1
# * "Video of exercise sessions week 37":"https://youtu.be/bK4AEcTu-oM"
* For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
!eblock