lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

changed variable names

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-07 19:55:00 +02:00
parent 831312ce60
commit f27b2c9260
17 changed files with 454 additions and 461 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week37/week37.do.txt
+46 -46
View File
@@ -543,8 +543,8 @@ function, which we want to minimize, can almost always be written as a
sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$,
!bt
\[
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\theta}).
\]
!et
@@ -555,8 +555,8 @@ This in turn means that the gradient can be
computed as a sum over $i$-gradients
!bt
\[
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}).
\]
!et
@@ -584,10 +584,10 @@ all data points with a sum over the data points in one the minibatches
picked at random in each gradient descent step
!bt
\[
\nabla_{\beta}
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
\nabla_{\theta}
C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta
c_i(\mathbf{x}_i, \mathbf{\theta}).
\]
!et
@@ -597,8 +597,8 @@ c_i(\mathbf{x}_i, \mathbf{\beta}).
Thus a gradient descent step now looks like
!bt
\[
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
\theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta})
\]
!et
@@ -647,7 +647,7 @@ is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the $\beta$ that
compare the values of the cost function and keep the $\theta$ that
gave the lowest value.
!split
@@ -668,10 +668,10 @@ for a discussion of different scaling functions for the learning rate.
As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.
In this way we can fix the number of epochs, compute $\beta$ and
In this way we can fix the number of epochs, compute $\theta$ and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final $\beta$ that gives the lowest value of the cost
pick the final $\theta$ that gives the lowest value of the cost
function.
!bc pycod
@@ -693,7 +693,7 @@ for epoch in range(1,n_epochs+1):
for i in range(m):
k = np.random.randint(m) #Pick the k-th minibatch at random
#Compute the gradient using the data in minibatch Bk
#Compute new suggestion for beta
#Compute new suggestion for theta
t = epoch*m+i
gamma_j = step_length(t,t0,t1)
j += 1
@@ -1005,10 +1005,10 @@ Addresses AdaGrad’s diminishing learning rate issue.
Uses a decaying average of squared gradients (instead of a cumulative sum):
!bt
\[
v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2,
v_t = \theta_2\, v_{t-1} + (1-\theta_2)\, (\nabla L(w_t))^2,
\]
!et
with $\beta_2$ typically $0.9$ (or $0.99$).
with $\theta_2$ typically $0.9$ (or $0.99$).
o Update: $w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t)$.
o Recent gradients have more weight, so $v_t$ adapts to the current landscape.
o Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
@@ -1039,13 +1039,13 @@ for RMS prop is given by
!bt
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\
\mathbf{s}_t &=\theta \mathbf{s}_{t-1} +(1-\theta)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
!et
where $\beta$ controls the averaging time of the second moment and is
typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate
where $\theta$ controls the averaging time of the second moment and is
typically taken to be about $\theta=0.9$, $\eta_t$ is a learning rate
typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a
small regularization constant to prevent divergences. Multiplication
and division by vectors is understood as an element-wise operation. It
@@ -1080,15 +1080,15 @@ again understood to be element-wise operations below)
!bt
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \bm{\mathbf{m}}_t \over \sqrt{\bm{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\end{align}
!et
where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and
where $\theta_1$ and $\theta_2$ set the memory lifetime of the first and
second moment and are typically taken to be $0.9$ and $0.99$
respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
@@ -1165,8 +1165,8 @@ import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
def CostOLS(beta):
return (1.0/n)*np.sum((y-X @ beta)**2)
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
@@ -1221,8 +1221,8 @@ import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
def CostOLS(beta):
return (1.0/n)*np.sum((y-X @ beta)**2)
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
@@ -1440,8 +1440,8 @@ import numpy as np
import autograd.numpy as np
from autograd import grad
def CostOLS(beta):
return (1.0/n)*np.sum((y-X @ beta)**2)
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
@@ -1449,24 +1449,24 @@ y = 4+3*x+5*x*x
X = np.c_[np.ones((n,1)), x, x*x]
XT_X = X.T @ X
beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print("Own inversion")
print(beta_linreg)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
# Note that here the Hessian does not depend on the parameters beta
# Note that here the Hessian does not depend on the parameters theta
invH = np.linalg.pinv(H)
beta = np.random.randn(3,1)
theta = np.random.randn(3,1)
Niterations = 5
# define the gradient
training_gradient = grad(CostOLS)
for iter in range(Niterations):
gradients = training_gradient(beta)
beta -= invH @ gradients
gradients = training_gradient(theta)
theta -= invH @ gradients
print(iter,gradients[0],gradients[1])
print("beta from own Newton code")
print(beta)
print("theta from own Newton code")
print(theta)
!ec
@@ -1626,9 +1626,9 @@ theta = np.random.randn(3,1)
# Value for learning rate
eta = 0.01
# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980
beta1 = 0.9
beta2 = 0.999
# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980
theta1 = 0.9
theta2 = 0.999
# Including AdaGrad parameter to avoid possible division by zero
delta = 1e-7
iter = 0
@@ -1642,10 +1642,10 @@ for epoch in range(n_epochs):
yi = y[random_index:random_index+M]
gradients = (1.0/M)*training_gradient(yi, xi, theta)
# Computing moments first
first_moment = beta1*first_moment + (1-beta1)*gradients
second_moment = beta2*second_moment+(1-beta2)*gradients*gradients
first_term = first_moment/(1.0-beta1**iter)
second_term = second_moment/(1.0-beta2**iter)
first_moment = theta1*first_moment + (1-theta1)*gradients
second_moment = theta2*second_moment+(1-theta2)*gradients*gradients
first_term = first_moment/(1.0-theta1**iter)
second_term = second_moment/(1.0-theta2**iter)
# Scaling with rho the new and the previous results
update = eta*first_term/(np.sqrt(second_term)+delta)
theta -= update
@@ -1662,7 +1662,7 @@ print(theta)
o Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)
o Work on project 1
# * "Video of exercise sessions week 37":"https://youtu.be/bK4AEcTu-oM"
* For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
!eblock