update

doc/src/week40/week40.do.txt

@@ -21,6 +21,45 @@ For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.
 "What is Stochastic Gradient Descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer"
 
 
+!split
+=====  Batches and mini-batches =====
+
+In gradient descent we compute the cost function and its gradient for all data points we have.
+
+In large-scale applications such as the "ILSVRC challenge":"https://www.image-net.org/challenges/LSVRC/", the
+training data can have on order of millions of examples. Hence, it
+seems wasteful to compute the full cost function over the entire
+training set in order to perform only a single parameter update. A
+very common approach to addressing this challenge is to compute the
+gradient over batches of the training data. For example, in current
+a typical batch could contain some thousand  examples from
+an  entire training set of several millions. This batch is then used to
+perform a parameter update.
+
+!split
+===== Stochastic Gradient Descent (SGD) =====
+
+In stochastic gradient descent, the extreme case is the case where we
+have only one batch, that is we include the whole data set.
+
+This process is called Stochastic Gradient
+Descent (SGD) (or also sometimes on-line gradient descent). This is
+relatively less common to see because in practice due to vectorized
+code optimizations it can be computationally much more efficient to
+evaluate the gradient for 100 examples, than the gradient for one
+example 100 times. Even though SGD technically refers to using a
+single example at a time to evaluate the gradient, you will hear
+people use the term SGD even when referring to mini-batch gradient
+descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD
+for “Batch gradient descent” are rare to see), where it is usually
+assumed that mini-batches are used. The size of the mini-batch is a
+hyperparameter but it is not very common to cross-validate or bootstrap it. It is
+usually based on memory constraints (if any), or set to some value,
+e.g. 32, 64 or 128. We use powers of 2 in practice because many
+vectorized operation implementations work faster when their inputs are
+sized in powers of 2.
+
+In our notes with  SGD we mean stochastic gradient descent with mini-batches.
 
 
 !split
@@ -59,6 +98,7 @@ $k=1,\cdots,n/M$.
 
 !split
 ===== SGD example =====
+
 As an example, suppose we have $10$ data points $(\mathbf{x}_1,\cdots, \mathbf{x}_{10})$ 
 and we choose to have $M=5$ minibathces,
 then each minibatch contains two data points. In particular we have
@@ -228,6 +268,8 @@ ypredict2 = Xnew.dot(theta_linreg)
 
 
 n_epochs = 50
+M = 10   #size of each minibatch
+m = int(n/M) #number of minibatches
 t0, t1 = 5, 50
 def learning_schedule(t):
     return t0/(t+t1)
@@ -256,7 +298,7 @@ plt.show()
 
 !ec
 
-_Challenge_: try to write a similar code for a Logistic Regression case.
+
 
 
 !split