lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-07 13:50:43 +02:00
parent edd76e3b6a
commit bbdc10814d
55 changed files with 4427 additions and 3647 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week37/week37.do.txt
+79 -64
View File
@@ -21,7 +21,7 @@ o More advanced updates of the learning rate: ADAgrad, RMSprop and ADAM
!split
===== Readings and Videos: =====
!bblock
o Recommended: Goodfellow et al, Deep Learning, introduction to gradient descent, see sections 4.3-4.5 at URL:"https://www.deeplearningbook.org/contents/numerical.html" and chapter 8.3-8.5 at URL::https://www.deeplearningbook.org/contents/optimization.html"
o Recommended: Goodfellow et al, Deep Learning, introduction to gradient descent, see sections 4.3-4.5 at URL:"https://www.deeplearningbook.org/contents/numerical.html" and chapter 8.3-8.5 at URL:"https://www.deeplearningbook.org/contents/optimization.html"
o Rashcka et al, pages 37-44 and pages 278-283 with focus on linear regression.
o Video on gradient descent at URL:"https://www.youtube.com/watch?v=sDv4f4s2SB8"
o Video on Stochastic gradient descent at URL:"https://www.youtube.com/watch?v=vMh0zPT0tLI"
@@ -452,13 +452,15 @@ pyplot.plot(solutions, scores, '.-', color='red')
pyplot.show()
!ec
!split
===== Overview video on Stochastic Gradient Descent =====
===== Overview video on Stochastic Gradient Descent (SGD) =====
"What is Stochastic Gradient Descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer"
There are several reasons for using stochastic gradient descent. Some of these are:
o Efficiency: Updates weights more frequently using a single or a small batch of samples, which speeds up convergence.
o Hopefully avoid Local Minima
o Memory Usage: Requires less memory compared to computing gradients for the entire dataset.
!split
===== Batches and mini-batches =====
@@ -474,6 +476,44 @@ gradient over batches of the training data. For example, a typical batch could c
an entire training set of several millions. This batch is then used to
perform a parameter update.
!split
===== Pros and cons =====
o Speed: SGD is faster than gradient descent because it uses only one training example per iteration, whereas gradient descent requires the entire dataset. This speed advantage becomes more significant as the size of the dataset increases.
o Convergence: Gradient descent has a more predictable convergence behaviour because it uses the average gradient of the entire dataset. In contrast, SGD’s convergence behaviour can be more erratic due to its random sampling of individual training examples.
o Memory: Gradient descent requires more memory than SGD because it must store the entire dataset for each iteration. SGD only needs to store the current training example, making it more memory-efficient.
!split
===== Convergence rates =====
o Stochastic Gradient Descent has a faster convergence rate due to the use of single training examples in each iteration.
o Gradient Descent as a slower convergence rate, as it uses the entire dataset for each iteration.
!split
===== Accuracy =====
In general, stochastic Gradient Descent is Less accurate than gradient
descent, as it calculates the gradient on single examples, which may
not accurately represent the overall dataset. Gradient Descent is
more accurate because it uses the average gradient calculated over the
entire dataset.
There are other disadvantages to using SGD. The main drawback is that
its convergence behaviour can be more erratic due to the random
sampling of individual training examples. This can lead to less
accurate results, as the algorithm may not converge to the true
minimum of the cost function. Additionally, the learning rate, which
determines the step size of each update to the model’s parameters,
must be carefully chosen to ensure convergence.
It is however the method of choice in deep learning algorithms where
SGD is often used in combination with other optimization techniques,
such as momentum or adaptive learning rates
!split
===== Stochastic Gradient Descent (SGD) =====
@@ -670,10 +710,6 @@ print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
!ec
!split
===== Code with a Number of Minibatches which varies =====
@@ -870,8 +906,8 @@ our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma
\end{align}
!et
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\gamma$.
One of the major advantages of NAG is that it allows for the use of a
larger learning rate than GDM for the same choice of $\gamma$.
!split
===== Second moment of the gradient =====
@@ -893,7 +929,7 @@ adaptively change the step size to match the landscape without paying
the steep computational price of calculating or approximating
Hessians.
Recently, a number of methods have been introduced that accomplish
During the last decade a number of methods have been introduced that accomplish
this by tracking not only the gradient, but also the second moment of
the gradient. These methods include AdaGrad, AdaDelta, Root Mean Squared Propagation (RMS-Prop), and
"ADAM":"https://arxiv.org/abs/1412.6980".
@@ -980,7 +1016,26 @@ update rule for this parameter is given by
The algorithms we have implemented are well described in the text by "Goodfellow, Bengio and Courville, chapter 8":"https://www.deeplearningbook.org/contents/optimization.html".
The codes which implement these algorithms are discussed below here.
The codes which implement these algorithms are discussed after our presentation of automatic differentiation.
===== AdaGrad algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" =====
FIGURE: [figures/adagrad.png, width=600 frac=0.8]
===== RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" =====
FIGURE: [figures/rmsprop.png, width=600 frac=0.8]
===== ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" =====
FIGURE: [figures/adam.png, width=600 frac=0.8]
!split
@@ -995,15 +1050,12 @@ The codes which implement these algorithms are discussed below here.
* _Adaptive optimization methods don't always have good generalization._ Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications.
!split
===== Sneaking in automatic differentiation using Autograd =====
We anticipate our discussions to come in connection with neural networks and automatic differentiation
by showing how we can use _autograd_ for the cases above. Later we will replace _autograd_ with _JAX_.
===== Sneaking in auotmatic differentiation using Autograd =====
We conclude the part on optmization by showing how we can make codes
for linear regression and logistic regression using _autograd_. The
first example shows results with ordinary leats squares.
!bc pycod
# Using Autograd to calculate gradients for OLS
@@ -1118,51 +1170,6 @@ print(theta)
!ec
!split
===== But none of these can compete with Newton's method =====
!bc pycod
# Using Newton's method
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
def CostOLS(beta):
return (1.0/n)*np.sum((y-X @ beta)**2)
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
XT_X = X.T @ X
beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print("Own inversion")
print(beta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
# Note that here the Hessian does not depend on the parameters beta
invH = np.linalg.pinv(H)
EigValues, EigVectors = np.linalg.eig(H)
print(f"Eigenvalues of Hessian Matrix:{EigValues}")
beta = np.random.randn(2,1)
Niterations = 5
# define the gradient
training_gradient = grad(CostOLS)
for iter in range(Niterations):
gradients = training_gradient(beta)
beta -= invH @ gradients
print(iter,gradients[0],gradients[1])
print("beta from own Newton code")
print(beta)
!ec
!split
===== Including Stochastic Gradient Descent with Autograd =====
In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using _autograd_.
@@ -1512,3 +1519,11 @@ o Work on project 1
# * "Video of exercise sessions week 37":"https://youtu.be/bK4AEcTu-oM"
* For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
!eblock