update of book

doc/LectureNotes/_build/html/week37.html

@@ -416,6 +416,18 @@ document.write(`
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#time-decay-rate">Time decay rate</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#code-with-a-number-of-minibatches-which-varies">Code with a Number of Minibatches which varies</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#replace-or-not">Replace or not</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison">SGD vs Full-Batch GD: Convergence Speed and Memory Comparison</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#theoretical-convergence-speed-and-convex-optimization">Theoretical Convergence Speed and convex optimization</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#strongly-convex-case">Strongly Convex Case</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#non-convex-problems">Non-Convex Problems</a></li>
+</ul>
+</li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-usage-and-scalability">Memory Usage and Scalability</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#empirical-evidence-convergence-time-and-memory-in-practice">Empirical Evidence: Convergence Time and Memory in Practice</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deep-neural-networks">Deep Neural Networks</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-constraints">Memory constraints</a></li>
+</ul>
+</li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#second-moment-of-the-gradient">Second moment of the gradient</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#challenge-choosing-a-fixed-learning-rate">Challenge: Choosing a Fixed Learning Rate</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#motivation-for-adaptive-step-sizes">Motivation for Adaptive Step Sizes</a></li>
@@ -1202,6 +1214,225 @@ mini-batches. The discussion
 <a class="reference external" href="https://sebastianraschka.com/faq/docs/sgd-methods.html">here</a> may be
 useful.</p>
 </section>
+<section id="sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison">
+<h2>SGD vs Full-Batch GD: Convergence Speed and Memory Comparison<a class="headerlink" href="#sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison" title="Link to this heading">#</a></h2>
+<section id="theoretical-convergence-speed-and-convex-optimization">
+<h3>Theoretical Convergence Speed and convex optimization<a class="headerlink" href="#theoretical-convergence-speed-and-convex-optimization" title="Link to this heading">#</a></h3>
+<p>Consider minimizing an empirical cost function</p>
+<div class="math notranslate nohighlight">
+\[
+C(\theta) =\frac{1}{N}\sum_{i=1}^N l_i(\theta),
+\]</div>
+<p>where each <span class="math notranslate nohighlight">\(l_i(\theta)\)</span> is a
+differentiable loss term. Gradient Descent (GD) updates parameters
+using the full gradient <span class="math notranslate nohighlight">\(\nabla C(\theta)\)</span>, while Stochastic Gradient
+Descent (SGD) uses a single sample (or mini-batch) gradient <span class="math notranslate nohighlight">\(\nabla
+l_i(\theta)\)</span> selected at random. In equation form, one GD step is:</p>
+<div class="math notranslate nohighlight">
+\[
+\theta_{t+1} = \theta_t-\eta \nabla C(\theta_t) =\theta_t -\eta \frac{1}{N}\sum_{i=1}^N \nabla l_i(\theta_t),
+\]</div>
+<p>whereas one SGD step is:</p>
+<div class="math notranslate nohighlight">
+\[
+\theta_{t+1} = \theta_t -\eta \nabla l_{i_t}(\theta_t),
+\]</div>
+<p>with <span class="math notranslate nohighlight">\(i_t\)</span> randomly chosen. On smooth convex problems, GD and SGD both
+converge to the global minimum, but their rates differ. GD can take
+larger, more stable steps since it uses the exact gradient, achieving
+an error that decreases on the order of <span class="math notranslate nohighlight">\(O(1/t)\)</span> per iteration for
+convex objectives (and even exponentially fast for strongly convex
+cases). In contrast, plain SGD has more variance in each step, leading
+to sublinear convergence in expectation – typically <span class="math notranslate nohighlight">\(O(1/\sqrt{t})\)</span>
+for general convex objectives (\thetaith appropriate diminishing step
+sizes) . Intuitively, GD’s trajectory is smoother and more
+predictable, while SGD’s path oscillates due to noise but costs far
+less per iteration, enabling many more updates in the same time.</p>
+</section>
+<section id="strongly-convex-case">
+<h3>Strongly Convex Case<a class="headerlink" href="#strongly-convex-case" title="Link to this heading">#</a></h3>
+<p>If <span class="math notranslate nohighlight">\(C(\theta)\)</span> is strongly convex and <span class="math notranslate nohighlight">\(L\)</span>-smooth (so GD enjoys linear
+convergence), the gap <span class="math notranslate nohighlight">\(C(\theta_t)-C(\theta^*)\)</span> for GD shrinks as</p>
+<div class="math notranslate nohighlight">
+\[
+C(\theta_t) - C(\theta^* ) \le \Big(1 - \frac{\mu}{L}\Big)^t [C(\theta_0)-C(\theta^*)],
+\]</div>
+<p>a geometric (linear) convergence per iteration . Achieving an
+<span class="math notranslate nohighlight">\(\epsilon\)</span>-accurate solution thus takes on the order of
+<span class="math notranslate nohighlight">\(\log(1/\epsilon)\)</span> iterations for GD. However, each GD iteration costs
+<span class="math notranslate nohighlight">\(O(N)\)</span> gradient evaluations. SGD cannot exploit strong convexity to
+obtain a linear rate – instead, with a properly decaying step size
+(e.g. <span class="math notranslate nohighlight">\(\eta_t = \frac{1}{\mu t}\)</span>) or iterate averaging, SGD attains an
+<span class="math notranslate nohighlight">\(O(1/t)\)</span> convergence rate in expectation . For example, one result
+of Moulines and  Bach 2011, see <a class="reference external" href="https://papers.nips.cc/paper_files/paper/2011/hash/40008b9a5380fcacce3976bf7c08af5b-Abstract.html">https://papers.nips.cc/paper_files/paper/2011/hash/40008b9a5380fcacce3976bf7c08af5b-Abstract.html</a> shows that with <span class="math notranslate nohighlight">\(\eta_t = \Theta(1/t)\)</span>,</p>
+<div class="math notranslate nohighlight">
+\[
+\mathbb{E}[C(\theta_t) - C(\theta^*)] = O(1/t),
+\]</div>
+<p>for strongly convex, smooth <span class="math notranslate nohighlight">\(F\)</span> . This <span class="math notranslate nohighlight">\(1/t\)</span> rate is slower per
+iteration than GD’s exponential decay, but each SGD iteration is <span class="math notranslate nohighlight">\(N\)</span>
+times cheaper. In fact, to reach error <span class="math notranslate nohighlight">\(\epsilon\)</span>, plain SGD needs on
+the order of <span class="math notranslate nohighlight">\(T=O(1/\epsilon)\)</span> iterations (sub-linear convergence),
+while GD needs <span class="math notranslate nohighlight">\(O(\log(1/\epsilon))\)</span> iterations. When accounting for
+cost-per-iteration, GD requires <span class="math notranslate nohighlight">\(O(N \log(1/\epsilon))\)</span> total gradient
+computations versus SGD’s <span class="math notranslate nohighlight">\(O(1/\epsilon)\)</span> single-sample
+computations. In large-scale regimes (huge <span class="math notranslate nohighlight">\(N\)</span>), SGD can be
+faster in wall-clock time because <span class="math notranslate nohighlight">\(N \log(1/\epsilon)\)</span> may far exceed
+<span class="math notranslate nohighlight">\(1/\epsilon\)</span> for reasonable accuracy levels. In other words,
+with millions of data points, one epoch of GD (one full gradient) is
+extremely costly, whereas SGD can make <span class="math notranslate nohighlight">\(N\)</span> cheap updates in the time
+GD makes one – often yielding a good solution faster in practice, even
+though SGD’s asymptotic error decays more slowly. As one lecture
+succinctly puts it: “SGD can be super effective in terms of iteration
+cost and memory, but SGD is slow to converge and can’t adapt to strong
+convexity” . Thus, the break-even point depends on <span class="math notranslate nohighlight">\(N\)</span> and the desired
+accuracy: for moderate accuracy on very large <span class="math notranslate nohighlight">\(N\)</span>, SGD’s cheaper
+updates win; for extremely high precision (very small <span class="math notranslate nohighlight">\(\epsilon\)</span>) on a
+modest <span class="math notranslate nohighlight">\(N\)</span>, GD’s fast convergence per step can be advantageous.</p>
+</section>
+<section id="non-convex-problems">
+<h3>Non-Convex Problems<a class="headerlink" href="#non-convex-problems" title="Link to this heading">#</a></h3>
+<p>In non-convex optimization (e.g. deep neural networks), neither GD nor
+SGD guarantees global minima, but SGD often displays faster progress
+in finding useful minima. Theoretical results here are weaker, usually
+showing convergence to a stationary point <span class="math notranslate nohighlight">\(\theta\)</span> (<span class="math notranslate nohighlight">\(|\nabla C|\)</span> is
+small) in expectation. For example, GD might require <span class="math notranslate nohighlight">\(O(1/\epsilon^2)\)</span>
+iterations to ensure <span class="math notranslate nohighlight">\(|\nabla C(\theta)| &lt; \epsilon\)</span>, and SGD typically has
+similar polynomial complexity (often worse due to gradient
+noise). However, a noteworthy difference is that SGD’s stochasticity
+can help escape saddle points or poor local minima. Random gradient
+fluctuations act like implicit noise, helping the iterate “jump” out
+of flat saddle regions where full-batch GD could stagnate . In fact,
+research has shown that adding noise to GD can guarantee escaping
+saddle points in polynomial time, and the inherent noise in SGD often
+serves this role. Empirically, this means SGD can sometimes find a
+lower loss basin faster, whereas full-batch GD might get “stuck” near
+saddle points or need a very small learning rate to navigate complex
+error surfaces . Overall, in modern high-dimensional machine learning,
+SGD (or mini-batch SGD) is the workhorse for large non-convex problems
+because it converges to good solutions much faster in practice,
+despite the lack of a linear convergence guarantee. Full-batch GD is
+rarely used on large neural networks, as it would require tiny steps
+to avoid divergence and is extremely slow per iteration .</p>
+</section>
+</section>
+<section id="memory-usage-and-scalability">
+<h2>Memory Usage and Scalability<a class="headerlink" href="#memory-usage-and-scalability" title="Link to this heading">#</a></h2>
+<p>A major advantage of SGD is its memory efficiency in handling large
+datasets. Full-batch GD requires access to the entire training set for
+each iteration, which often means the whole dataset (or a large
+subset) must reside in memory to compute <span class="math notranslate nohighlight">\(\nabla C(\theta)\)</span> . This results
+in memory usage that scales linearly with the dataset size <span class="math notranslate nohighlight">\(N\)</span>. For
+instance, if each training sample is large (e.g. high-dimensional
+features), computing a full gradient may require storing a substantial
+portion of the data or all intermediate gradients until they are
+aggregated. In contrast, SGD needs only a single (or a small
+mini-batch of) training example(s) in memory at any time . The
+algorithm processes one sample (or mini-batch) at a time and
+immediately updates the model, discarding that sample before moving to
+the next. This streaming approach means that memory footprint is
+essentially independent of <span class="math notranslate nohighlight">\(N\)</span> (apart from storing the model
+parameters themselves). As one source notes, gradient descent
+“requires more memory than SGD” because it “must store the entire
+dataset for each iteration,” whereas SGD “only needs to store the
+current training example” . In practical terms, if you have a dataset
+of size, say, 1 million examples, full-batch GD would need memory for
+all million every step, while SGD could be implemented to load just
+one example at a time – a crucial benefit if data are too large to fit
+in RAM or GPU memory. This scalability makes SGD suitable for
+large-scale learning: as long as you can stream data from disk, SGD
+can handle arbitrarily large datasets with fixed memory. In fact, SGD
+“does not need to remember which examples were visited” in the past,
+allowing it to run in an online fashion on infinite data streams
+. Full-batch GD, on the other hand, would require multiple passes
+through a giant dataset per update (or a complex distributed memory
+system), which is often infeasible.</p>
+<p>There is also a secondary memory effect: computing a full-batch
+gradient in deep learning requires storing all intermediate
+activations for backpropagation across the entire batch. A very large
+batch (approaching the full dataset) might exhaust GPU memory due to
+the need to hold activation gradients for thousands or millions of
+examples simultaneously. SGD/minibatches mitigate this by splitting
+the workload – e.g. with a mini-batch of size 32 or 256, memory use
+stays bounded, whereas a full-batch (size = <span class="math notranslate nohighlight">\(N\)</span>) forward/backward pass
+could not even be executed if <span class="math notranslate nohighlight">\(N\)</span> is huge. Techniques like gradient
+accumulation exist to simulate large-batch GD by summing many
+small-batch gradients – but these still process data in manageable
+chunks to avoid memory overflow. In summary, memory complexity for GD
+grows with <span class="math notranslate nohighlight">\(N\)</span>, while for SGD it remains <span class="math notranslate nohighlight">\(O(1)\)</span> w.r.t. dataset size
+(only the model and perhaps a mini-batch reside in memory) . This is a
+key reason why batch GD “does not scale” to very large data and why
+virtually all large-scale machine learning algorithms rely on
+stochastic or mini-batch methods.</p>
+</section>
+<section id="empirical-evidence-convergence-time-and-memory-in-practice">
+<h2>Empirical Evidence: Convergence Time and Memory in Practice<a class="headerlink" href="#empirical-evidence-convergence-time-and-memory-in-practice" title="Link to this heading">#</a></h2>
+<p>Empirical studies strongly support the theoretical trade-offs
+above. In large-scale machine learning tasks, SGD often converges to a
+good solution much faster in wall-clock time than full-batch GD, and
+it uses far less memory. For example, Bottou &amp; Bousquet (2008)
+analyzed learning time under a fixed computational budget and
+concluded that when data is abundant, it’s better to use a faster
+(even if less precise) optimization method to process more examples in
+the same time . This analysis showed that for large-scale problems,
+processing more data with SGD yields lower error than spending the
+time to do exact (batch) optimization on fewer data . In other words,
+if you have a time budget, it’s often optimal to accept slightly
+slower convergence per step (as with SGD) in exchange for being able
+to use many more training samples in that time. This phenomenon is
+borne out by experiments:</p>
+<section id="deep-neural-networks">
+<h3>Deep Neural Networks<a class="headerlink" href="#deep-neural-networks" title="Link to this heading">#</a></h3>
+<p>In modern deep learning, full-batch GD is so slow that it is rarely
+attempted; instead, mini-batch SGD is standard. A recent study
+demonstrated that it is possible to train a ResNet-50 on ImageNet
+using full-batch gradient descent, but it required careful tuning
+(e.g. gradient clipping, tiny learning rates) and vast computational
+resources – and even then, each full-batch update was extremely
+expensive.</p>
+<p>Using a huge batch
+(closer to full GD) tends to slow down convergence if the learning
+rate is not scaled up, and often encounters optimization difficulties
+(plateaus) that small batches avoid.
+Empirically, small or medium
+batch SGD finds minima in fewer clock hours because it can rapidly
+loop over the data with gradient noise aiding exploration.</p>
+</section>
+<section id="memory-constraints">
+<h3>Memory constraints<a class="headerlink" href="#memory-constraints" title="Link to this heading">#</a></h3>
+<p>From a memory standpoint, practitioners note that batch GD becomes
+infeasible on large data. For example, if one tried to do full-batch
+training on a dataset that doesn’t fit in RAM or GPU memory, the
+program would resort to heavy disk I/O or simply crash. SGD
+circumvents this by processing mini-batches. Even in cases where data
+does fit in memory, using a full batch can spike memory usage due to
+storing all gradients. One empirical observation is that mini-batch
+training has a “lower, fluctuating usage pattern” of memory, whereas
+full-batch loading “quickly consumes memory (often exceeding limits)”
+. This is especially relevant for graph neural networks or other
+models where a “batch” may include a huge chunk of a graph: full-batch
+gradient computation can exhaust GPU memory, whereas mini-batch
+methods keep memory usage manageable .</p>
+<p>In summary, SGD converges faster than full-batch GD in terms of actual
+training time for large-scale problems, provided we measure
+convergence as reaching a good-enough solution. Theoretical bounds
+show SGD needs more iterations, but because it performs many more
+updates per unit time (and requires far less memory), it often
+achieves lower loss in a given time frame than GD. Full-batch GD might
+take slightly fewer iterations in theory, but each iteration is so
+costly that it is “slower… especially for large datasets” . Meanwhile,
+memory scaling strongly favors SGD: GD’s memory cost grows with
+dataset size, making it impractical beyond a point, whereas SGD’s
+memory use is modest and mostly constant w.r.t. <span class="math notranslate nohighlight">\(N\)</span> . These
+differences have made SGD (and mini-batch variants) the de facto
+choice for training large machine learning models, from logistic
+regression on millions of examples to deep neural networks with
+billions of parameters. The consensus in both research and practice is
+that for large-scale or high-dimensional tasks, SGD-type methods
+converge quicker per unit of computation and handle memory constraints
+better than standard full-batch gradient descent .</p>
+</section>
+</section>
 <section id="second-moment-of-the-gradient">
 <h2>Second moment of the gradient<a class="headerlink" href="#second-moment-of-the-gradient" title="Link to this heading">#</a></h2>
 <p>In stochastic gradient descent, with and without momentum, we still
@@ -2560,6 +2791,18 @@ centered matrix and/or vector that enter the fitting procedure.</p>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#time-decay-rate">Time decay rate</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#code-with-a-number-of-minibatches-which-varies">Code with a Number of Minibatches which varies</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#replace-or-not">Replace or not</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison">SGD vs Full-Batch GD: Convergence Speed and Memory Comparison</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#theoretical-convergence-speed-and-convex-optimization">Theoretical Convergence Speed and convex optimization</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#strongly-convex-case">Strongly Convex Case</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#non-convex-problems">Non-Convex Problems</a></li>
+</ul>
+</li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-usage-and-scalability">Memory Usage and Scalability</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#empirical-evidence-convergence-time-and-memory-in-practice">Empirical Evidence: Convergence Time and Memory in Practice</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deep-neural-networks">Deep Neural Networks</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-constraints">Memory constraints</a></li>
+</ul>
+</li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#second-moment-of-the-gradient">Second moment of the gradient</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#challenge-choosing-a-fixed-learning-rate">Challenge: Choosing a Fixed Learning Rate</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#motivation-for-adaptive-step-sizes">Motivation for Adaptive Step Sizes</a></li>