lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9e321dd76bc0373845a8f3818048d42837d3e532
FYS-STK4155/doc/LectureNotes/_build/html/week37.html
T
Morten Hjorth-Jensen 385d3865db updated lecturenotes
2025-10-13 09:15:15 +02:00

2920 lines
161 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en" data-content_root="./" >
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Week 37: Gradient descent methods &#8212; Applied Data Analysis and Machine Learning</title>
<script data-cfasync="false">
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
</script>
<!-- Loaded before other Sphinx assets -->
<link href="_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" />
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" />
<link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" />
<script src="_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/documentation_options.js?v=9eb32ce0"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/clipboard.min.js?v=a7894cd8"></script>
<script src="_static/copybutton.js?v=f281be69"></script>
<script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js?v=4a39c7ea"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script src="_static/design-tabs.js?v=f930bc37"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script async="async" src="_static/sphinx-thebe.js?v=c100c467"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script>DOCUMENTATION_OPTIONS.pagename = 'week37';</script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Exercises week 38" href="exercisesweek38.html" />
<link rel="prev" title="Exercises week 37" href="exercisesweek37.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
</head>
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
<div id="pst-scroll-pixel-helper"></div>
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
<input type="checkbox"
class="sidebar-toggle"
id="pst-primary-sidebar-checkbox"/>
<label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label>
<input type="checkbox"
class="sidebar-toggle"
id="pst-secondary-sidebar-checkbox"/>
<label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label>
<div class="search-button__wrapper">
<div class="search-button__overlay"></div>
<div class="search-button__search-container">
<form class="bd-search d-flex align-items-center"
action="search.html"
method="get">
<i class="fa-solid fa-magnifying-glass"></i>
<input type="search"
class="form-control"
name="q"
id="search-input"
placeholder="Search this book..."
aria-label="Search this book..."
autocomplete="off"
autocorrect="off"
autocapitalize="off"
spellcheck="false"/>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
</form></div>
</div>
<div class="pst-async-banner-revealer d-none">
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
</div>
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
</header>
<div class="bd-container">
<div class="bd-container__inner bd-page-width">
<div class="bd-sidebar-primary bd-sidebar">
<div class="sidebar-header-items sidebar-primary__section">
</div>
<div class="sidebar-primary-items__start sidebar-primary__section">
<div class="sidebar-primary-item">
<a class="navbar-brand logo" href="intro.html">
<img src="_static/logo.png" class="logo__image only-light" alt="Applied Data Analysis and Machine Learning - Home"/>
<script>document.write(`<img src="_static/logo.png" class="logo__image only-dark" alt="Applied Data Analysis and Machine Learning - Home"/>`);</script>
</a></div>
<div class="sidebar-primary-item">
<script>
document.write(`
<button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
`);
</script></div>
<div class="sidebar-primary-item"><nav class="bd-links bd-docs-nav" aria-label="Main">
<div class="bd-toc-item navbar-nav active">
<ul class="nav bd-sidenav bd-sidenav__home-link">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">About the course</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="schedule.html">Course setting</a></li>
<li class="toctree-l1"><a class="reference internal" href="teachers.html">Teachers and Grading</a></li>
<li class="toctree-l1"><a class="reference internal" href="textbooks.html">Textbooks</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Review of Statistics with Resampling Techniques and Linear Algebra</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="statistics.html">1. Elements of Probability Theory and Statistical Data Analysis</a></li>
<li class="toctree-l1"><a class="reference internal" href="linalg.html">2. Linear Algebra, Handling of Arrays and more Python Features</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">From Regression to Support Vector Machines</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter1.html">3. Linear Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter2.html">4. Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter3.html">5. Resampling Methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter4.html">6. Logistic Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapteroptimization.html">7. Optimization, the central part of any Machine Learning algortithm</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter5.html">8. Support Vector Machines, overarching aims</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Decision Trees, Ensemble Methods and Boosting</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter6.html">9. Decision trees, overarching aims</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter7.html">10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Dimensionality Reduction</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter8.html">11. Basic ideas of the Principal Component Analysis (PCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="clustering.html">12. Clustering and Unsupervised Learning</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Deep Learning Methods</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter9.html">13. Neural networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter10.html">14. Building a Feed Forward Neural Network</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter11.html">15. Solving Differential Equations with Deep Learning</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter12.html">16. Convolutional Neural Networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter13.html">17. Recurrent neural networks: Overarching view</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek35.html">Exercises week 35</a></li>
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Week 37: Gradient descent methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek38.html">Exercises week 38</a></li>
<li class="toctree-l1"><a class="reference internal" href="week38.html">Week 38: Statistical analysis, bias-variance tradeoff and resampling methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek39.html">Exercises week 39</a></li>
<li class="toctree-l1"><a class="reference internal" href="week39.html">Week 39: Resampling methods and logistic regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="week40.html">Week 40: Gradient descent methods (continued) and start Neural networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="week41.html">Week 41 Neural networks and constructing a neural network code</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek41.html">Exercises week 41</a></li>
<li class="toctree-l1"><a class="reference internal" href="week42.html">Week 42 Constructing a Neural Network code with examples</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek42.html">Exercises week 42</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Projects</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="project1.html">Project 1 on Machine Learning, deadline October 6 (midnight), 2025</a></li>
<li class="toctree-l1"><a class="reference internal" href="project2.html">Project 2 on Machine Learning, deadline November 10 (Midnight)</a></li>
</ul>
</div>
</nav></div>
</div>
<div class="sidebar-primary-items__end sidebar-primary__section">
</div>
<div id="rtd-footer-container"></div>
</div>
<main id="main-content" class="bd-main" role="main">
<div class="sbt-scroll-pixel-helper"></div>
<div class="bd-content">
<div class="bd-article-container">
<div class="bd-header-article d-print-none">
<div class="header-article-items header-article__inner">
<div class="header-article-items__start">
<div class="header-article-item"><button class="sidebar-toggle primary-toggle btn btn-sm" title="Toggle primary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-bars"></span>
</button></div>
</div>
<div class="header-article-items__end">
<div class="header-article-item">
<div class="article-header-buttons">
<div class="dropdown dropdown-download-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Download this page">
<i class="fas fa-download"></i>
</button>
<ul class="dropdown-menu">
<li><a href="_sources/week37.ipynb" target="_blank"
class="btn btn-sm btn-download-source-button dropdown-item"
title="Download source file"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file"></i>
</span>
<span class="btn__text-container">.ipynb</span>
</a>
</li>
<li>
<button onclick="window.print()"
class="btn btn-sm btn-download-pdf-button dropdown-item"
title="Print to PDF"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file-pdf"></i>
</span>
<span class="btn__text-container">.pdf</span>
</button>
</li>
</ul>
</div>
<button onclick="toggleFullScreen()"
class="btn btn-sm btn-fullscreen-button"
title="Fullscreen mode"
data-bs-placement="bottom" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-expand"></i>
</span>
</button>
<script>
document.write(`
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i>
</button>
`);
</script>
<script>
document.write(`
<button class="btn btn-sm pst-navbar-icon search-button search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass fa-lg"></i>
</button>
`);
</script>
<button class="sidebar-toggle secondary-toggle btn btn-sm" title="Toggle secondary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-list"></span>
</button>
</div></div>
</div>
</div>
</div>
<div id="jb-print-docs-body" class="onlyprint">
<h1>Week 37: Gradient descent methods</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#plans-for-week-37-lecture-monday">Plans for week 37, lecture Monday</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#readings-and-videos">Readings and Videos:</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#material-for-lecture-monday-september-8">Material for lecture Monday September 8</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-and-revisiting-ordinary-least-squares-from-last-week">Gradient descent and revisiting Ordinary Least Squares from last week</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-example">Gradient descent example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-derivative-of-the-cost-loss-function">The derivative of the cost/loss function</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix">The Hessian matrix</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-program">Simple program</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Gradient Descent Example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-and-ridge">Gradient descent and Ridge</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix-for-ridge-regression">The Hessian matrix for Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#using-gradient-descent-methods-limitations">Using gradient descent methods, limitations</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#momentum-based-gd">Momentum based GD</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#improving-gradient-descent-with-momentum">Improving gradient descent with momentum</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#same-code-but-now-with-momentum-gradient-descent">Same code but now with momentum gradient descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#overview-video-on-stochastic-gradient-descent-sgd">Overview video on Stochastic Gradient Descent (SGD)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#batches-and-mini-batches">Batches and mini-batches</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#pros-and-cons">Pros and cons</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#convergence-rates">Convergence rates</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#accuracy">Accuracy</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#stochastic-gradient-descent">Stochastic Gradient Descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#computation-of-gradients">Computation of gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sgd-example">SGD example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-gradient-step">The gradient step</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-example-code">Simple example code</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#when-do-we-stop">When do we stop?</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#slightly-different-approach">Slightly different approach</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#time-decay-rate">Time decay rate</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#code-with-a-number-of-minibatches-which-varies">Code with a Number of Minibatches which varies</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#replace-or-not">Replace or not</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison">SGD vs Full-Batch GD: Convergence Speed and Memory Comparison</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#theoretical-convergence-speed-and-convex-optimization">Theoretical Convergence Speed and convex optimization</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#strongly-convex-case">Strongly Convex Case</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#non-convex-problems">Non-Convex Problems</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-usage-and-scalability">Memory Usage and Scalability</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#empirical-evidence-convergence-time-and-memory-in-practice">Empirical Evidence: Convergence Time and Memory in Practice</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deep-neural-networks">Deep Neural Networks</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-constraints">Memory constraints</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#second-moment-of-the-gradient">Second moment of the gradient</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#challenge-choosing-a-fixed-learning-rate">Challenge: Choosing a Fixed Learning Rate</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#motivation-for-adaptive-step-sizes">Motivation for Adaptive Step Sizes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adagrad-algorithm-taken-from-goodfellow-et-al">AdaGrad algorithm, taken from Goodfellow et al</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#derivation-of-the-adagrad-algorithm">Derivation of the AdaGrad Algorithm</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adagrad-update-rule-derivation">AdaGrad Update Rule Derivation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adagrad-properties">AdaGrad Properties</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rmsprop-adaptive-learning-rates">RMSProp: Adaptive Learning Rates</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rmsprop-algorithm-taken-from-goodfellow-et-al">RMSProp algorithm, taken from Goodfellow et al</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-optimizer">Adam Optimizer</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">ADAM optimizer</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-combine-momentum-and-rmsprop">Why Combine Momentum and RMSProp?</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-exponential-moving-averages-moments">Adam: Exponential Moving Averages (Moments)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-bias-correction">Adam: Bias Correction</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-update-rule-derivation">Adam: Update Rule Derivation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-vs-adagrad-and-rmsprop">Adam vs. AdaGrad and RMSProp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adaptivity-across-dimensions">Adaptivity Across Dimensions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-algorithm-taken-from-goodfellow-et-al">ADAM algorithm, taken from Goodfellow et al</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#algorithms-and-codes-for-adagrad-rmsprop-and-adam">Algorithms and codes for Adagrad, RMSprop and Adam</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#practical-tips">Practical tips</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sneaking-in-automatic-differentiation-using-autograd">Sneaking in automatic differentiation using Autograd</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">Same code but now with momentum gradient descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">Including Stochastic Gradient Descent with Autograd</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id4">Same code but now with momentum gradient descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#but-none-of-these-can-compete-with-newton-s-method">But none of these can compete with Newton’s method</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#similar-second-order-function-now-problem-but-now-with-adagrad">Similar (second order function now) problem but now with AdaGrad</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent">RMSprop for adaptive learning rate with Stochastic Gradient Descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#and-finally-adam">And finally ADAM</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#material-for-the-lab-sessions">Material for the lab sessions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reminder-on-different-scaling-methods">Reminder on different scaling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#functionality-in-scikit-learn">Functionality in Scikit-Learn</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#more-preprocessing">More preprocessing</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#frequently-used-scaling-functions">Frequently used scaling functions</a></li>
</ul>
</nav>
</div>
</div>
</div>
<div id="searchbox"></div>
<article class="bd-article">
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html week37.do.txt --no_mako -->
<!-- dom:TITLE: Week 37: Gradient descent methods --><section class="tex2jax_ignore mathjax_ignore" id="week-37-gradient-descent-methods">
<h1>Week 37: Gradient descent methods<a class="headerlink" href="#week-37-gradient-descent-methods" title="Link to this heading">#</a></h1>
<p><strong>Morten Hjorth-Jensen</strong>, Department of Physics, University of Oslo, Norway</p>
<p>Date: <strong>September 8-12, 2025</strong></p>
<!-- todo add link to videos and add link to Van Wieringens notes --><section id="plans-for-week-37-lecture-monday">
<h2>Plans for week 37, lecture Monday<a class="headerlink" href="#plans-for-week-37-lecture-monday" title="Link to this heading">#</a></h2>
<p><strong>Plans and material for the lecture on Monday September 8.</strong></p>
<p>The family of gradient descent methods</p>
<ol class="arabic simple">
<li><p>Plain gradient descent (constant learning rate), reminder from last week with examples using OLS and Ridge</p></li>
<li><p>Improving gradient descent with momentum</p></li>
<li><p>Introducing stochastic gradient descent</p></li>
<li><p>More advanced updates of the learning rate: ADAgrad, RMSprop and ADAM</p></li>
<li><p><a class="reference external" href="https://youtu.be/SuxK68tj-V8">Video of Lecture</a></p></li>
<li><p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2025/FYSSTKweek37.pdf">Whiteboard notes</a></p></li>
</ol>
</section>
<section id="readings-and-videos">
<h2>Readings and Videos:<a class="headerlink" href="#readings-and-videos" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Recommended: Goodfellow et al, Deep Learning, introduction to gradient descent, see sections 4.3-4.5 at <a class="reference external" href="https://www.deeplearningbook.org/contents/numerical.html">https://www.deeplearningbook.org/contents/numerical.html</a> and chapter 8.3-8.5 at <a class="reference external" href="https://www.deeplearningbook.org/contents/optimization.html">https://www.deeplearningbook.org/contents/optimization.html</a></p></li>
<li><p>Rashcka et al, pages 37-44 and pages 278-283 with focus on linear regression.</p></li>
<li><p>Video on gradient descent at <a class="reference external" href="https://www.youtube.com/watch?v=sDv4f4s2SB8">https://www.youtube.com/watch?v=sDv4f4s2SB8</a></p></li>
<li><p>Video on Stochastic gradient descent at <a class="reference external" href="https://www.youtube.com/watch?v=vMh0zPT0tLI">https://www.youtube.com/watch?v=vMh0zPT0tLI</a></p></li>
</ol>
</section>
<section id="material-for-lecture-monday-september-8">
<h2>Material for lecture Monday September 8<a class="headerlink" href="#material-for-lecture-monday-september-8" title="Link to this heading">#</a></h2>
</section>
<section id="gradient-descent-and-revisiting-ordinary-least-squares-from-last-week">
<h2>Gradient descent and revisiting Ordinary Least Squares from last week<a class="headerlink" href="#gradient-descent-and-revisiting-ordinary-least-squares-from-last-week" title="Link to this heading">#</a></h2>
<p>Last week we started with linear regression as a case study for the gradient descent
methods. Linear regression is a great test case for the gradient
descent methods discussed in the lectures since it has several
desirable properties such as:</p>
<ol class="arabic simple">
<li><p>An analytical solution (recall homework sets for week 35).</p></li>
<li><p>The gradient can be computed analytically.</p></li>
<li><p>The cost function is convex which guarantees that gradient descent converges for small enough learning rates</p></li>
</ol>
<p>We revisit an example similar to what we had in the first homework set. We have a function of the type</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
</pre></div>
</div>
</div>
</div>
<p>with <span class="math notranslate nohighlight">\(x_i \in [0,1] \)</span> is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution <span class="math notranslate nohighlight">\(\cal {N}(0,1)\)</span>.
The linear regression model is given by</p>
<div class="math notranslate nohighlight">
\[
h_\theta(x) = \boldsymbol{y} = \theta_0 + \theta_1 x,
\]</div>
<p>such that</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{y}_i = \theta_0 + \theta_1 x_i.
\]</div>
</section>
<section id="gradient-descent-example">
<h2>Gradient descent example<a class="headerlink" href="#gradient-descent-example" title="Link to this heading">#</a></h2>
<p>Let <span class="math notranslate nohighlight">\(\mathbf{y} = (y_1,\cdots,y_n)^T\)</span>, <span class="math notranslate nohighlight">\(\mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T\)</span> and <span class="math notranslate nohighlight">\(\theta = (\theta_0, \theta_1)^T\)</span></p>
<p>It is convenient to write <span class="math notranslate nohighlight">\(\mathbf{\boldsymbol{y}} = X\theta\)</span> where <span class="math notranslate nohighlight">\(X \in \mathbb{R}^{100 \times 2} \)</span> is the design matrix given by (we keep the intercept here)</p>
<div class="math notranslate nohighlight">
\[\begin{split}
X \equiv \begin{bmatrix}
1 &amp; x_1 \\
\vdots &amp; \vdots \\
1 &amp; x_{100} &amp; \\
\end{bmatrix}.
\end{split}\]</div>
<p>The cost/loss/risk function is given by</p>
<div class="math notranslate nohighlight">
\[
C(\theta) = \frac{1}{n}||X\theta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\theta_0 + \theta_1 x_i)^2 - 2 y_i (\theta_0 + \theta_1 x_i) + y_i^2\right]
\]</div>
<p>and we want to find <span class="math notranslate nohighlight">\(\theta\)</span> such that <span class="math notranslate nohighlight">\(C(\theta)\)</span> is minimized.</p>
</section>
<section id="the-derivative-of-the-cost-loss-function">
<h2>The derivative of the cost/loss function<a class="headerlink" href="#the-derivative-of-the-cost-loss-function" title="Link to this heading">#</a></h2>
<p>Computing <span class="math notranslate nohighlight">\(\partial C(\theta) / \partial \theta_0\)</span> and <span class="math notranslate nohighlight">\(\partial C(\theta) / \partial \theta_1\)</span> we can show that the gradient can be written as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\nabla_{\theta} C(\theta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\theta_0+\theta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\theta_0+\theta_1x_i)-y_ix_i\right) \\
\end{bmatrix} = \frac{2}{n}X^T(X\theta - \mathbf{y}),
\end{split}\]</div>
<p>where <span class="math notranslate nohighlight">\(X\)</span> is the design matrix defined above.</p>
</section>
<section id="the-hessian-matrix">
<h2>The Hessian matrix<a class="headerlink" href="#the-hessian-matrix" title="Link to this heading">#</a></h2>
<p>The Hessian matrix of <span class="math notranslate nohighlight">\(C(\theta)\)</span> is given by</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\theta)}{\partial \theta_0^2} &amp; \frac{\partial^2 C(\theta)}{\partial \theta_0 \partial \theta_1} \\
\frac{\partial^2 C(\theta)}{\partial \theta_0 \partial \theta_1} &amp; \frac{\partial^2 C(\theta)}{\partial \theta_1^2} &amp; \\
\end{bmatrix} = \frac{2}{n}X^T X.
\end{split}\]</div>
<p>This result implies that <span class="math notranslate nohighlight">\(C(\theta)\)</span> is a convex function since the matrix <span class="math notranslate nohighlight">\(X^T X\)</span> always is positive semi-definite.</p>
</section>
<section id="simple-program">
<h2>Simple program<a class="headerlink" href="#simple-program" title="Link to this heading">#</a></h2>
<p>We can now write a program that minimizes <span class="math notranslate nohighlight">\(C(\theta)\)</span> using the gradient descent method with a constant learning rate <span class="math notranslate nohighlight">\(\eta\)</span> according to</p>
<div class="math notranslate nohighlight">
\[
\theta_{k+1} = \theta_k - \eta \nabla_\theta C(\theta_k), \ k=0,1,\cdots
\]</div>
<p>We can use the expression we computed for the gradient and let use a
<span class="math notranslate nohighlight">\(\theta_0\)</span> be chosen randomly and let <span class="math notranslate nohighlight">\(\eta = 0.001\)</span>. Stop iterating
when <span class="math notranslate nohighlight">\(||\nabla_\theta C(\theta_k) || \leq \epsilon = 10^{-8}\)</span>. <strong>Note that the code below does not include the latter stop criterion</strong>.</p>
<p>And finally we can compare our solution for <span class="math notranslate nohighlight">\(\theta\)</span> with the analytic result given by
<span class="math notranslate nohighlight">\(\theta= (X^TX)^{-1} X^T \mathbf{y}\)</span>.</p>
</section>
<section id="id1">
<h2>Gradient Descent Example<a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
<p>Here our simple example</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>%matplotlib inline
# Importing various packages
from random import random, seed
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import sys
# the number of datapoints
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
# Hessian matrix
H = (2.0/n)* X.T @ X
# Get the eigenvalues
EigValues, EigVectors = np.linalg.eig(H)
print(f&quot;Eigenvalues of Hessian Matrix:{EigValues}&quot;)
theta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y
print(theta_linreg)
theta = np.random.randn(2,1)
eta = 1.0/np.max(EigValues)
Niterations = 1000
for iter in range(Niterations):
gradient = (2.0/n)*X.T @ (X @ theta-y)
theta -= eta*gradient
print(theta)
xnew = np.array([[0],[2]])
xbnew = np.c_[np.ones((2,1)), xnew]
ypredict = xbnew.dot(theta)
ypredict2 = xbnew.dot(theta_linreg)
plt.plot(xnew, ypredict, &quot;r-&quot;)
plt.plot(xnew, ypredict2, &quot;b-&quot;)
plt.plot(x, y ,&#39;ro&#39;)
plt.axis([0,2.0,0, 15.0])
plt.xlabel(r&#39;$x$&#39;)
plt.ylabel(r&#39;$y$&#39;)
plt.title(r&#39;Gradient descent example&#39;)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="gradient-descent-and-ridge">
<h2>Gradient descent and Ridge<a class="headerlink" href="#gradient-descent-and-ridge" title="Link to this heading">#</a></h2>
<p>We have also discussed Ridge regression where the loss function contains a regularized term given by the <span class="math notranslate nohighlight">\(L_2\)</span> norm of <span class="math notranslate nohighlight">\(\theta\)</span>,</p>
<div class="math notranslate nohighlight">
\[
C_{\text{ridge}}(\theta) = \frac{1}{n}||X\theta -\mathbf{y}||^2 + \lambda ||\theta||^2, \ \lambda \geq 0.
\]</div>
<p>In order to minimize <span class="math notranslate nohighlight">\(C_{\text{ridge}}(\theta)\)</span> using GD we adjust the gradient as follows</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\nabla_\theta C_{\text{ridge}}(\theta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\theta_0+\theta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\theta_0+\theta_1x_i)-y_ix_i\right) \\
\end{bmatrix} + 2\lambda\begin{bmatrix} \theta_0 \\ \theta_1\end{bmatrix} = 2 (\frac{1}{n}X^T(X\theta - \mathbf{y})+\lambda \theta).
\end{split}\]</div>
<p>We can easily extend our program to minimize <span class="math notranslate nohighlight">\(C_{\text{ridge}}(\theta)\)</span> using gradient descent and compare with the analytical solution given by</p>
<div class="math notranslate nohighlight">
\[
\theta_{\text{ridge}} = \left(X^T X + n\lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}.
\]</div>
</section>
<section id="the-hessian-matrix-for-ridge-regression">
<h2>The Hessian matrix for Ridge Regression<a class="headerlink" href="#the-hessian-matrix-for-ridge-regression" title="Link to this heading">#</a></h2>
<p>The Hessian matrix of Ridge Regression for our simple example is given by</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\theta)}{\partial \theta_0^2} &amp; \frac{\partial^2 C(\theta)}{\partial \theta_0 \partial \theta_1} \\
\frac{\partial^2 C(\theta)}{\partial \theta_0 \partial \theta_1} &amp; \frac{\partial^2 C(\theta)}{\partial \theta_1^2} &amp; \\
\end{bmatrix} = \frac{2}{n}X^T X+2\lambda\boldsymbol{I}.
\end{split}\]</div>
<p>This implies that the Hessian matrix is positive definite, hence the stationary point is a
minimum.
Note that the Ridge cost function is convex being a sum of two convex
functions. Therefore, the stationary point is a global
minimum of this function.</p>
</section>
<section id="program-example-for-gradient-descent-with-ridge-regression">
<h2>Program example for gradient descent with Ridge Regression<a class="headerlink" href="#program-example-for-gradient-descent-with-ridge-regression" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from random import random, seed
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import sys
# the number of datapoints
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
XT_X = X.T @ X
#Ridge parameter lambda
lmbda = 0.001
Id = n*lmbda* np.eye(XT_X.shape[0])
# Hessian matrix
H = (2.0/n)* XT_X+2*lmbda* np.eye(XT_X.shape[0])
# Get the eigenvalues
EigValues, EigVectors = np.linalg.eig(H)
print(f&quot;Eigenvalues of Hessian Matrix:{EigValues}&quot;)
theta_linreg = np.linalg.inv(XT_X+Id) @ X.T @ y
print(theta_linreg)
# Start plain gradient descent
theta = np.random.randn(2,1)
eta = 1.0/np.max(EigValues)
Niterations = 100
for iter in range(Niterations):
gradients = 2.0/n*X.T @ (X @ (theta)-y)+2*lmbda*theta
theta -= eta*gradients
print(theta)
ypredict = X @ theta
ypredict2 = X @ theta_linreg
plt.plot(x, ypredict, &quot;r-&quot;)
plt.plot(x, ypredict2, &quot;b-&quot;)
plt.plot(x, y ,&#39;ro&#39;)
plt.axis([0,2.0,0, 15.0])
plt.xlabel(r&#39;$x$&#39;)
plt.ylabel(r&#39;$y$&#39;)
plt.title(r&#39;Gradient descent example for Ridge&#39;)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="using-gradient-descent-methods-limitations">
<h2>Using gradient descent methods, limitations<a class="headerlink" href="#using-gradient-descent-methods-limitations" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p><strong>Gradient descent (GD) finds local minima of our function</strong>. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our cost/loss/risk function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.</p></li>
<li><p><strong>GD is sensitive to initial conditions</strong>. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.</p></li>
<li><p><strong>Gradients are computationally expensive to calculate for large datasets</strong>. In many cases in statistics and ML, the cost/loss/risk function is a sum of terms, with one term for each data point. For example, in linear regression, <span class="math notranslate nohighlight">\(E \propto \sum_{i=1}^n (y_i - \mathbf{w}^T\cdot\mathbf{x}_i)^2\)</span>; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over <em>all</em> <span class="math notranslate nohighlight">\(n\)</span> data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called “mini batches”. This has the added benefit of introducing stochasticity into our algorithm.</p></li>
<li><p><strong>GD is very sensitive to choices of learning rates</strong>. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would <em>adaptively</em> choose the learning rates to match the landscape.</p></li>
<li><p><strong>GD treats all directions in parameter space uniformly.</strong> Another major drawback of GD is that unlike Newton’s method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.</p></li>
<li><p>GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.</p></li>
</ul>
</section>
<section id="momentum-based-gd">
<h2>Momentum based GD<a class="headerlink" href="#momentum-based-gd" title="Link to this heading">#</a></h2>
<p>We discuss here some simple examples where we introduce what is called
‘memory’about previous steps, or what is normally called momentum
gradient descent.
For the mathematical details, see whiteboad notes from lecture on September 8, 2025.</p>
</section>
<section id="improving-gradient-descent-with-momentum">
<h2>Improving gradient descent with momentum<a class="headerlink" href="#improving-gradient-descent-with-momentum" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from numpy import asarray
from numpy import arange
from numpy.random import rand
from numpy.random import seed
from matplotlib import pyplot
# objective function
def objective(x):
return x**2.0
# derivative of objective function
def derivative(x):
return x * 2.0
# gradient descent algorithm
def gradient_descent(objective, derivative, bounds, n_iter, step_size):
# track all solutions
solutions, scores = list(), list()
# generate an initial point
solution = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])
# run the gradient descent
for i in range(n_iter):
# calculate gradient
gradient = derivative(solution)
# take a step
solution = solution - step_size * gradient
# evaluate candidate point
solution_eval = objective(solution)
# store solution
solutions.append(solution)
scores.append(solution_eval)
# report progress
print(&#39;&gt;%d f(%s) = %.5f&#39; % (i, solution, solution_eval))
return [solutions, scores]
# seed the pseudo random number generator
seed(4)
# define range for input
bounds = asarray([[-1.0, 1.0]])
# define the total iterations
n_iter = 30
# define the step size
step_size = 0.1
# perform the gradient descent search
solutions, scores = gradient_descent(objective, derivative, bounds, n_iter, step_size)
# sample input range uniformly at 0.1 increments
inputs = arange(bounds[0,0], bounds[0,1]+0.1, 0.1)
# compute targets
results = objective(inputs)
# create a line plot of input vs result
pyplot.plot(inputs, results)
# plot the solutions found
pyplot.plot(solutions, scores, &#39;.-&#39;, color=&#39;red&#39;)
# show the plot
pyplot.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="same-code-but-now-with-momentum-gradient-descent">
<h2>Same code but now with momentum gradient descent<a class="headerlink" href="#same-code-but-now-with-momentum-gradient-descent" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from numpy import asarray
from numpy import arange
from numpy.random import rand
from numpy.random import seed
from matplotlib import pyplot
# objective function
def objective(x):
return x**2.0
# derivative of objective function
def derivative(x):
return x * 2.0
# gradient descent algorithm
def gradient_descent(objective, derivative, bounds, n_iter, step_size, momentum):
# track all solutions
solutions, scores = list(), list()
# generate an initial point
solution = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])
# keep track of the change
change = 0.0
# run the gradient descent
for i in range(n_iter):
# calculate gradient
gradient = derivative(solution)
# calculate update
new_change = step_size * gradient + momentum * change
# take a step
solution = solution - new_change
# save the change
change = new_change
# evaluate candidate point
solution_eval = objective(solution)
# store solution
solutions.append(solution)
scores.append(solution_eval)
# report progress
print(&#39;&gt;%d f(%s) = %.5f&#39; % (i, solution, solution_eval))
return [solutions, scores]
# seed the pseudo random number generator
seed(4)
# define range for input
bounds = asarray([[-1.0, 1.0]])
# define the total iterations
n_iter = 30
# define the step size
step_size = 0.1
# define momentum
momentum = 0.3
# perform the gradient descent search with momentum
solutions, scores = gradient_descent(objective, derivative, bounds, n_iter, step_size, momentum)
# sample input range uniformly at 0.1 increments
inputs = arange(bounds[0,0], bounds[0,1]+0.1, 0.1)
# compute targets
results = objective(inputs)
# create a line plot of input vs result
pyplot.plot(inputs, results)
# plot the solutions found
pyplot.plot(solutions, scores, &#39;.-&#39;, color=&#39;red&#39;)
# show the plot
pyplot.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="overview-video-on-stochastic-gradient-descent-sgd">
<h2>Overview video on Stochastic Gradient Descent (SGD)<a class="headerlink" href="#overview-video-on-stochastic-gradient-descent-sgd" title="Link to this heading">#</a></h2>
<p><a class="reference external" href="https://www.youtube.com/watch?v=vMh0zPT0tLI&amp;amp;ab_channel=StatQuestwithJoshStarmer">What is Stochastic Gradient Descent</a>
There are several reasons for using stochastic gradient descent. Some of these are:</p>
<ol class="arabic simple">
<li><p>Efficiency: Updates weights more frequently using a single or a small batch of samples, which speeds up convergence.</p></li>
<li><p>Hopefully avoid Local Minima</p></li>
<li><p>Memory Usage: Requires less memory compared to computing gradients for the entire dataset.</p></li>
</ol>
</section>
<section id="batches-and-mini-batches">
<h2>Batches and mini-batches<a class="headerlink" href="#batches-and-mini-batches" title="Link to this heading">#</a></h2>
<p>In gradient descent we compute the cost function and its gradient for all data points we have.</p>
<p>In large-scale applications such as the <a class="reference external" href="https://www.image-net.org/challenges/LSVRC/">ILSVRC challenge</a>, the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.</p>
</section>
<section id="pros-and-cons">
<h2>Pros and cons<a class="headerlink" href="#pros-and-cons" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Speed: SGD is faster than gradient descent because it uses only one training example per iteration, whereas gradient descent requires the entire dataset. This speed advantage becomes more significant as the size of the dataset increases.</p></li>
<li><p>Convergence: Gradient descent has a more predictable convergence behaviour because it uses the average gradient of the entire dataset. In contrast, SGD’s convergence behaviour can be more erratic due to its random sampling of individual training examples.</p></li>
<li><p>Memory: Gradient descent requires more memory than SGD because it must store the entire dataset for each iteration. SGD only needs to store the current training example, making it more memory-efficient.</p></li>
</ol>
</section>
<section id="convergence-rates">
<h2>Convergence rates<a class="headerlink" href="#convergence-rates" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Stochastic Gradient Descent has a faster convergence rate due to the use of single training examples in each iteration.</p></li>
<li><p>Gradient Descent as a slower convergence rate, as it uses the entire dataset for each iteration.</p></li>
</ol>
</section>
<section id="accuracy">
<h2>Accuracy<a class="headerlink" href="#accuracy" title="Link to this heading">#</a></h2>
<p>In general, stochastic Gradient Descent is Less accurate than gradient
descent, as it calculates the gradient on single examples, which may
not accurately represent the overall dataset. Gradient Descent is
more accurate because it uses the average gradient calculated over the
entire dataset.</p>
<p>There are other disadvantages to using SGD. The main drawback is that
its convergence behaviour can be more erratic due to the random
sampling of individual training examples. This can lead to less
accurate results, as the algorithm may not converge to the true
minimum of the cost function. Additionally, the learning rate, which
determines the step size of each update to the model’s parameters,
must be carefully chosen to ensure convergence.</p>
<p>It is however the method of choice in deep learning algorithms where
SGD is often used in combination with other optimization techniques,
such as momentum or adaptive learning rates</p>
</section>
<section id="stochastic-gradient-descent-sgd">
<h2>Stochastic Gradient Descent (SGD)<a class="headerlink" href="#stochastic-gradient-descent-sgd" title="Link to this heading">#</a></h2>
<p>In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.</p>
<p>This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD
for “Batch gradient descent” are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.</p>
<p>In our notes with SGD we mean stochastic gradient descent with mini-batches.</p>
</section>
<section id="stochastic-gradient-descent">
<h2>Stochastic Gradient Descent<a class="headerlink" href="#stochastic-gradient-descent" title="Link to this heading">#</a></h2>
<p>Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.</p>
<p>The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over <span class="math notranslate nohighlight">\(n\)</span> data points <span class="math notranslate nohighlight">\(\{\mathbf{x}_i\}_{i=1}^n\)</span>,</p>
<div class="math notranslate nohighlight">
\[
C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\theta}).
\]</div>
</section>
<section id="computation-of-gradients">
<h2>Computation of gradients<a class="headerlink" href="#computation-of-gradients" title="Link to this heading">#</a></h2>
<p>This in turn means that the gradient can be
computed as a sum over <span class="math notranslate nohighlight">\(i\)</span>-gradients</p>
<div class="math notranslate nohighlight">
\[
\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}).
\]</div>
<p>Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are <span class="math notranslate nohighlight">\(n\)</span>
data points and the size of each minibatch is <span class="math notranslate nohighlight">\(M\)</span>, there will be <span class="math notranslate nohighlight">\(n/M\)</span>
minibatches. We denote these minibatches by <span class="math notranslate nohighlight">\(B_k\)</span> where
<span class="math notranslate nohighlight">\(k=1,\cdots,n/M\)</span>.</p>
</section>
<section id="sgd-example">
<h2>SGD example<a class="headerlink" href="#sgd-example" title="Link to this heading">#</a></h2>
<p>As an example, suppose we have <span class="math notranslate nohighlight">\(10\)</span> data points <span class="math notranslate nohighlight">\((\mathbf{x}_1,\cdots, \mathbf{x}_{10})\)</span>
and we choose to have <span class="math notranslate nohighlight">\(M=5\)</span> minibathces,
then each minibatch contains two data points. In particular we have
<span class="math notranslate nohighlight">\(B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
(\mathbf{x}_9,\mathbf{x}_{10})\)</span>. Note that if you choose <span class="math notranslate nohighlight">\(M=1\)</span> you
have only a single batch with all data points and on the other extreme,
you may choose <span class="math notranslate nohighlight">\(M=n\)</span> resulting in a minibatch for each datapoint, i.e
<span class="math notranslate nohighlight">\(B_k = \mathbf{x}_k\)</span>.</p>
<p>The idea is now to approximate the gradient by replacing the sum over
all data points with a sum over the data points in one the minibatches
picked at random in each gradient descent step</p>
<div class="math notranslate nohighlight">
\[
\nabla_{\theta}
C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta
c_i(\mathbf{x}_i, \mathbf{\theta}).
\]</div>
</section>
<section id="the-gradient-step">
<h2>The gradient step<a class="headerlink" href="#the-gradient-step" title="Link to this heading">#</a></h2>
<p>Thus a gradient descent step now looks like</p>
<div class="math notranslate nohighlight">
\[
\theta_{j+1} = \theta_j - \eta_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i,
\mathbf{\theta})
\]</div>
<p>where <span class="math notranslate nohighlight">\(k\)</span> is picked at random with equal
probability from <span class="math notranslate nohighlight">\([1,n/M]\)</span>. An iteration over the number of
minibathces (n/M) is commonly referred to as an epoch. Thus it is
typical to choose a number of epochs and for each epoch iterate over
the number of minibatches, as exemplified in the code below.</p>
</section>
<section id="simple-example-code">
<h2>Simple example code<a class="headerlink" href="#simple-example-code" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
n = 100 #100 datapoints
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
n_epochs = 10 #number of epochs
j = 0
for epoch in range(1,n_epochs+1):
for i in range(m):
k = np.random.randint(m) #Pick the k-th minibatch at random
#Compute the gradient using the data in minibatch Bk
#Compute new suggestion for
j += 1
</pre></div>
</div>
</div>
</div>
<p>Taking the gradient only on a subset of the data has two important
benefits. First, it introduces randomness which decreases the chance
that our opmization scheme gets stuck in a local minima. Second, if
the size of the minibatches are small relative to the number of
datapoints (<span class="math notranslate nohighlight">\(M &lt; n\)</span>), the computation of the gradient is much
cheaper since we sum over the datapoints in the <span class="math notranslate nohighlight">\(k-th\)</span> minibatch and not
all <span class="math notranslate nohighlight">\(n\)</span> datapoints.</p>
</section>
<section id="when-do-we-stop">
<h2>When do we stop?<a class="headerlink" href="#when-do-we-stop" title="Link to this heading">#</a></h2>
<p>A natural question is when do we stop the search for a new minimum?
One possibility is to compute the full gradient after a given number
of epochs and check if the norm of the gradient is smaller than some
threshold and stop if true. However, the condition that the gradient
is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the <span class="math notranslate nohighlight">\(\theta\)</span> that
gave the lowest value.</p>
</section>
<section id="slightly-different-approach">
<h2>Slightly different approach<a class="headerlink" href="#slightly-different-approach" title="Link to this heading">#</a></h2>
<p>Another approach is to let the step length <span class="math notranslate nohighlight">\(\eta_j\)</span> depend on the
number of epochs in such a way that it becomes very small after a
reasonable time such that we do not move at all. Such approaches are
also called scaling. There are many such ways to <a class="reference external" href="https://towardsdatascience.com/gradient-descent-the-learning-rate-and-the-importance-of-feature-scaling-6c0b416596e1">scale the learning
rate</a>
and <a class="reference external" href="https://www.jmlr.org/papers/volume23/20-1258/20-1258.pdf">discussions here</a>. See
also
<a class="reference external" href="https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1">https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1</a>
for a discussion of different scaling functions for the learning rate.</p>
</section>
<section id="time-decay-rate">
<h2>Time decay rate<a class="headerlink" href="#time-decay-rate" title="Link to this heading">#</a></h2>
<p>As an example, let <span class="math notranslate nohighlight">\(e = 0,1,2,3,\cdots\)</span> denote the current epoch and let <span class="math notranslate nohighlight">\(t_0, t_1 &gt; 0\)</span> be two fixed numbers. Furthermore, let <span class="math notranslate nohighlight">\(t = e \cdot m + i\)</span> where <span class="math notranslate nohighlight">\(m\)</span> is the number of minibatches and <span class="math notranslate nohighlight">\(i=0,\cdots,m-1\)</span>. Then the function $<span class="math notranslate nohighlight">\(\eta_j(t; t_0, t_1) = \frac{t_0}{t+t_1} \)</span><span class="math notranslate nohighlight">\( goes to zero as the number of epochs gets large. I.e. we start with a step length \)</span>\eta_j (0; t_0, t_1) = t_0/t_1<span class="math notranslate nohighlight">\( which decays in *time* \)</span>t$.</p>
<p>In this way we can fix the number of epochs, compute <span class="math notranslate nohighlight">\(\theta\)</span> and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final <span class="math notranslate nohighlight">\(\theta\)</span> that gives the lowest value of the cost
function.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
def step_length(t,t0,t1):
return t0/(t+t1)
n = 100 #100 datapoints
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
n_epochs = 500 #number of epochs
t0 = 1.0
t1 = 10
eta_j = t0/t1
j = 0
for epoch in range(1,n_epochs+1):
for i in range(m):
k = np.random.randint(m) #Pick the k-th minibatch at random
#Compute the gradient using the data in minibatch Bk
#Compute new suggestion for theta
t = epoch*m+i
eta_j = step_length(t,t0,t1)
j += 1
print(&quot;eta_j after %d epochs: %g&quot; % (n_epochs,eta_j))
</pre></div>
</div>
</div>
</div>
</section>
<section id="code-with-a-number-of-minibatches-which-varies">
<h2>Code with a Number of Minibatches which varies<a class="headerlink" href="#code-with-a-number-of-minibatches-which-varies" title="Link to this heading">#</a></h2>
<p>In the code here we vary the number of mini-batches.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Importing various packages
from math import exp, sqrt
from random import random, seed
import numpy as np
import matplotlib.pyplot as plt
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
XT_X = X.T @ X
theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
EigValues, EigVectors = np.linalg.eig(H)
print(f&quot;Eigenvalues of Hessian Matrix:{EigValues}&quot;)
theta = np.random.randn(2,1)
eta = 1.0/np.max(EigValues)
Niterations = 1000
for iter in range(Niterations):
gradients = 2.0/n*X.T @ ((X @ theta)-y)
theta -= eta*gradients
print(&quot;theta from own gd&quot;)
print(theta)
xnew = np.array([[0],[2]])
Xnew = np.c_[np.ones((2,1)), xnew]
ypredict = Xnew.dot(theta)
ypredict2 = Xnew.dot(theta_linreg)
n_epochs = 50
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
t0, t1 = 5, 50
def learning_schedule(t):
return t0/(t+t1)
theta = np.random.randn(2,1)
for epoch in range(n_epochs):
# Can you figure out a better way of setting up the contributions to each batch?
for i in range(m):
random_index = M*np.random.randint(m)
xi = X[random_index:random_index+M]
yi = y[random_index:random_index+M]
gradients = (2.0/M)* xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
print(&quot;theta from own sdg&quot;)
print(theta)
plt.plot(xnew, ypredict, &quot;r-&quot;)
plt.plot(xnew, ypredict2, &quot;b-&quot;)
plt.plot(x, y ,&#39;ro&#39;)
plt.axis([0,2.0,0, 15.0])
plt.xlabel(r&#39;$x$&#39;)
plt.ylabel(r&#39;$y$&#39;)
plt.title(r&#39;Random numbers &#39;)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="replace-or-not">
<h2>Replace or not<a class="headerlink" href="#replace-or-not" title="Link to this heading">#</a></h2>
<p>In the above code, we have use replacement in setting up the
mini-batches. The discussion
<a class="reference external" href="https://sebastianraschka.com/faq/docs/sgd-methods.html">here</a> may be
useful.</p>
</section>
<section id="sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison">
<h2>SGD vs Full-Batch GD: Convergence Speed and Memory Comparison<a class="headerlink" href="#sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison" title="Link to this heading">#</a></h2>
<section id="theoretical-convergence-speed-and-convex-optimization">
<h3>Theoretical Convergence Speed and convex optimization<a class="headerlink" href="#theoretical-convergence-speed-and-convex-optimization" title="Link to this heading">#</a></h3>
<p>Consider minimizing an empirical cost function</p>
<div class="math notranslate nohighlight">
\[
C(\theta) =\frac{1}{N}\sum_{i=1}^N l_i(\theta),
\]</div>
<p>where each <span class="math notranslate nohighlight">\(l_i(\theta)\)</span> is a
differentiable loss term. Gradient Descent (GD) updates parameters
using the full gradient <span class="math notranslate nohighlight">\(\nabla C(\theta)\)</span>, while Stochastic Gradient
Descent (SGD) uses a single sample (or mini-batch) gradient <span class="math notranslate nohighlight">\(\nabla
l_i(\theta)\)</span> selected at random. In equation form, one GD step is:</p>
<div class="math notranslate nohighlight">
\[
\theta_{t+1} = \theta_t-\eta \nabla C(\theta_t) =\theta_t -\eta \frac{1}{N}\sum_{i=1}^N \nabla l_i(\theta_t),
\]</div>
<p>whereas one SGD step is:</p>
<div class="math notranslate nohighlight">
\[
\theta_{t+1} = \theta_t -\eta \nabla l_{i_t}(\theta_t),
\]</div>
<p>with <span class="math notranslate nohighlight">\(i_t\)</span> randomly chosen. On smooth convex problems, GD and SGD both
converge to the global minimum, but their rates differ. GD can take
larger, more stable steps since it uses the exact gradient, achieving
an error that decreases on the order of <span class="math notranslate nohighlight">\(O(1/t)\)</span> per iteration for
convex objectives (and even exponentially fast for strongly convex
cases). In contrast, plain SGD has more variance in each step, leading
to sublinear convergence in expectation – typically <span class="math notranslate nohighlight">\(O(1/\sqrt{t})\)</span>
for general convex objectives (\thetaith appropriate diminishing step
sizes) . Intuitively, GD’s trajectory is smoother and more
predictable, while SGD’s path oscillates due to noise but costs far
less per iteration, enabling many more updates in the same time.</p>
</section>
<section id="strongly-convex-case">
<h3>Strongly Convex Case<a class="headerlink" href="#strongly-convex-case" title="Link to this heading">#</a></h3>
<p>If <span class="math notranslate nohighlight">\(C(\theta)\)</span> is strongly convex and <span class="math notranslate nohighlight">\(L\)</span>-smooth (so GD enjoys linear
convergence), the gap <span class="math notranslate nohighlight">\(C(\theta_t)-C(\theta^*)\)</span> for GD shrinks as</p>
<div class="math notranslate nohighlight">
\[
C(\theta_t) - C(\theta^* ) \le \Big(1 - \frac{\mu}{L}\Big)^t [C(\theta_0)-C(\theta^*)],
\]</div>
<p>a geometric (linear) convergence per iteration . Achieving an
<span class="math notranslate nohighlight">\(\epsilon\)</span>-accurate solution thus takes on the order of
<span class="math notranslate nohighlight">\(\log(1/\epsilon)\)</span> iterations for GD. However, each GD iteration costs
<span class="math notranslate nohighlight">\(O(N)\)</span> gradient evaluations. SGD cannot exploit strong convexity to
obtain a linear rate – instead, with a properly decaying step size
(e.g. <span class="math notranslate nohighlight">\(\eta_t = \frac{1}{\mu t}\)</span>) or iterate averaging, SGD attains an
<span class="math notranslate nohighlight">\(O(1/t)\)</span> convergence rate in expectation . For example, one result
of Moulines and Bach 2011, see <a class="reference external" href="https://papers.nips.cc/paper_files/paper/2011/hash/40008b9a5380fcacce3976bf7c08af5b-Abstract.html">https://papers.nips.cc/paper_files/paper/2011/hash/40008b9a5380fcacce3976bf7c08af5b-Abstract.html</a> shows that with <span class="math notranslate nohighlight">\(\eta_t = \Theta(1/t)\)</span>,</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}[C(\theta_t) - C(\theta^*)] = O(1/t),
\]</div>
<p>for strongly convex, smooth <span class="math notranslate nohighlight">\(F\)</span> . This <span class="math notranslate nohighlight">\(1/t\)</span> rate is slower per
iteration than GD’s exponential decay, but each SGD iteration is <span class="math notranslate nohighlight">\(N\)</span>
times cheaper. In fact, to reach error <span class="math notranslate nohighlight">\(\epsilon\)</span>, plain SGD needs on
the order of <span class="math notranslate nohighlight">\(T=O(1/\epsilon)\)</span> iterations (sub-linear convergence),
while GD needs <span class="math notranslate nohighlight">\(O(\log(1/\epsilon))\)</span> iterations. When accounting for
cost-per-iteration, GD requires <span class="math notranslate nohighlight">\(O(N \log(1/\epsilon))\)</span> total gradient
computations versus SGD’s <span class="math notranslate nohighlight">\(O(1/\epsilon)\)</span> single-sample
computations. In large-scale regimes (huge <span class="math notranslate nohighlight">\(N\)</span>), SGD can be
faster in wall-clock time because <span class="math notranslate nohighlight">\(N \log(1/\epsilon)\)</span> may far exceed
<span class="math notranslate nohighlight">\(1/\epsilon\)</span> for reasonable accuracy levels. In other words,
with millions of data points, one epoch of GD (one full gradient) is
extremely costly, whereas SGD can make <span class="math notranslate nohighlight">\(N\)</span> cheap updates in the time
GD makes one – often yielding a good solution faster in practice, even
though SGD’s asymptotic error decays more slowly. As one lecture
succinctly puts it: “SGD can be super effective in terms of iteration
cost and memory, but SGD is slow to converge and can’t adapt to strong
convexity” . Thus, the break-even point depends on <span class="math notranslate nohighlight">\(N\)</span> and the desired
accuracy: for moderate accuracy on very large <span class="math notranslate nohighlight">\(N\)</span>, SGD’s cheaper
updates win; for extremely high precision (very small <span class="math notranslate nohighlight">\(\epsilon\)</span>) on a
modest <span class="math notranslate nohighlight">\(N\)</span>, GD’s fast convergence per step can be advantageous.</p>
</section>
<section id="non-convex-problems">
<h3>Non-Convex Problems<a class="headerlink" href="#non-convex-problems" title="Link to this heading">#</a></h3>
<p>In non-convex optimization (e.g. deep neural networks), neither GD nor
SGD guarantees global minima, but SGD often displays faster progress
in finding useful minima. Theoretical results here are weaker, usually
showing convergence to a stationary point <span class="math notranslate nohighlight">\(\theta\)</span> (<span class="math notranslate nohighlight">\(|\nabla C|\)</span> is
small) in expectation. For example, GD might require <span class="math notranslate nohighlight">\(O(1/\epsilon^2)\)</span>
iterations to ensure <span class="math notranslate nohighlight">\(|\nabla C(\theta)| &lt; \epsilon\)</span>, and SGD typically has
similar polynomial complexity (often worse due to gradient
noise). However, a noteworthy difference is that SGD’s stochasticity
can help escape saddle points or poor local minima. Random gradient
fluctuations act like implicit noise, helping the iterate “jump” out
of flat saddle regions where full-batch GD could stagnate . In fact,
research has shown that adding noise to GD can guarantee escaping
saddle points in polynomial time, and the inherent noise in SGD often
serves this role. Empirically, this means SGD can sometimes find a
lower loss basin faster, whereas full-batch GD might get “stuck” near
saddle points or need a very small learning rate to navigate complex
error surfaces . Overall, in modern high-dimensional machine learning,
SGD (or mini-batch SGD) is the workhorse for large non-convex problems
because it converges to good solutions much faster in practice,
despite the lack of a linear convergence guarantee. Full-batch GD is
rarely used on large neural networks, as it would require tiny steps
to avoid divergence and is extremely slow per iteration .</p>
</section>
</section>
<section id="memory-usage-and-scalability">
<h2>Memory Usage and Scalability<a class="headerlink" href="#memory-usage-and-scalability" title="Link to this heading">#</a></h2>
<p>A major advantage of SGD is its memory efficiency in handling large
datasets. Full-batch GD requires access to the entire training set for
each iteration, which often means the whole dataset (or a large
subset) must reside in memory to compute <span class="math notranslate nohighlight">\(\nabla C(\theta)\)</span> . This results
in memory usage that scales linearly with the dataset size <span class="math notranslate nohighlight">\(N\)</span>. For
instance, if each training sample is large (e.g. high-dimensional
features), computing a full gradient may require storing a substantial
portion of the data or all intermediate gradients until they are
aggregated. In contrast, SGD needs only a single (or a small
mini-batch of) training example(s) in memory at any time . The
algorithm processes one sample (or mini-batch) at a time and
immediately updates the model, discarding that sample before moving to
the next. This streaming approach means that memory footprint is
essentially independent of <span class="math notranslate nohighlight">\(N\)</span> (apart from storing the model
parameters themselves). As one source notes, gradient descent
“requires more memory than SGD” because it “must store the entire
dataset for each iteration,” whereas SGD “only needs to store the
current training example” . In practical terms, if you have a dataset
of size, say, 1 million examples, full-batch GD would need memory for
all million every step, while SGD could be implemented to load just
one example at a time – a crucial benefit if data are too large to fit
in RAM or GPU memory. This scalability makes SGD suitable for
large-scale learning: as long as you can stream data from disk, SGD
can handle arbitrarily large datasets with fixed memory. In fact, SGD
“does not need to remember which examples were visited” in the past,
allowing it to run in an online fashion on infinite data streams
. Full-batch GD, on the other hand, would require multiple passes
through a giant dataset per update (or a complex distributed memory
system), which is often infeasible.</p>
<p>There is also a secondary memory effect: computing a full-batch
gradient in deep learning requires storing all intermediate
activations for backpropagation across the entire batch. A very large
batch (approaching the full dataset) might exhaust GPU memory due to
the need to hold activation gradients for thousands or millions of
examples simultaneously. SGD/minibatches mitigate this by splitting
the workload – e.g. with a mini-batch of size 32 or 256, memory use
stays bounded, whereas a full-batch (size = <span class="math notranslate nohighlight">\(N\)</span>) forward/backward pass
could not even be executed if <span class="math notranslate nohighlight">\(N\)</span> is huge. Techniques like gradient
accumulation exist to simulate large-batch GD by summing many
small-batch gradients – but these still process data in manageable
chunks to avoid memory overflow. In summary, memory complexity for GD
grows with <span class="math notranslate nohighlight">\(N\)</span>, while for SGD it remains <span class="math notranslate nohighlight">\(O(1)\)</span> w.r.t. dataset size
(only the model and perhaps a mini-batch reside in memory) . This is a
key reason why batch GD “does not scale” to very large data and why
virtually all large-scale machine learning algorithms rely on
stochastic or mini-batch methods.</p>
</section>
<section id="empirical-evidence-convergence-time-and-memory-in-practice">
<h2>Empirical Evidence: Convergence Time and Memory in Practice<a class="headerlink" href="#empirical-evidence-convergence-time-and-memory-in-practice" title="Link to this heading">#</a></h2>
<p>Empirical studies strongly support the theoretical trade-offs
above. In large-scale machine learning tasks, SGD often converges to a
good solution much faster in wall-clock time than full-batch GD, and
it uses far less memory. For example, Bottou &amp; Bousquet (2008)
analyzed learning time under a fixed computational budget and
concluded that when data is abundant, it’s better to use a faster
(even if less precise) optimization method to process more examples in
the same time . This analysis showed that for large-scale problems,
processing more data with SGD yields lower error than spending the
time to do exact (batch) optimization on fewer data . In other words,
if you have a time budget, it’s often optimal to accept slightly
slower convergence per step (as with SGD) in exchange for being able
to use many more training samples in that time. This phenomenon is
borne out by experiments:</p>
<section id="deep-neural-networks">
<h3>Deep Neural Networks<a class="headerlink" href="#deep-neural-networks" title="Link to this heading">#</a></h3>
<p>In modern deep learning, full-batch GD is so slow that it is rarely
attempted; instead, mini-batch SGD is standard. A recent study
demonstrated that it is possible to train a ResNet-50 on ImageNet
using full-batch gradient descent, but it required careful tuning
(e.g. gradient clipping, tiny learning rates) and vast computational
resources – and even then, each full-batch update was extremely
expensive.</p>
<p>Using a huge batch
(closer to full GD) tends to slow down convergence if the learning
rate is not scaled up, and often encounters optimization difficulties
(plateaus) that small batches avoid.
Empirically, small or medium
batch SGD finds minima in fewer clock hours because it can rapidly
loop over the data with gradient noise aiding exploration.</p>
</section>
<section id="memory-constraints">
<h3>Memory constraints<a class="headerlink" href="#memory-constraints" title="Link to this heading">#</a></h3>
<p>From a memory standpoint, practitioners note that batch GD becomes
infeasible on large data. For example, if one tried to do full-batch
training on a dataset that doesn’t fit in RAM or GPU memory, the
program would resort to heavy disk I/O or simply crash. SGD
circumvents this by processing mini-batches. Even in cases where data
does fit in memory, using a full batch can spike memory usage due to
storing all gradients. One empirical observation is that mini-batch
training has a “lower, fluctuating usage pattern” of memory, whereas
full-batch loading “quickly consumes memory (often exceeding limits)”
. This is especially relevant for graph neural networks or other
models where a “batch” may include a huge chunk of a graph: full-batch
gradient computation can exhaust GPU memory, whereas mini-batch
methods keep memory usage manageable .</p>
<p>In summary, SGD converges faster than full-batch GD in terms of actual
training time for large-scale problems, provided we measure
convergence as reaching a good-enough solution. Theoretical bounds
show SGD needs more iterations, but because it performs many more
updates per unit time (and requires far less memory), it often
achieves lower loss in a given time frame than GD. Full-batch GD might
take slightly fewer iterations in theory, but each iteration is so
costly that it is “slower… especially for large datasets” . Meanwhile,
memory scaling strongly favors SGD: GD’s memory cost grows with
dataset size, making it impractical beyond a point, whereas SGD’s
memory use is modest and mostly constant w.r.t. <span class="math notranslate nohighlight">\(N\)</span> . These
differences have made SGD (and mini-batch variants) the de facto
choice for training large machine learning models, from logistic
regression on millions of examples to deep neural networks with
billions of parameters. The consensus in both research and practice is
that for large-scale or high-dimensional tasks, SGD-type methods
converge quicker per unit of computation and handle memory constraints
better than standard full-batch gradient descent .</p>
</section>
</section>
<section id="second-moment-of-the-gradient">
<h2>Second moment of the gradient<a class="headerlink" href="#second-moment-of-the-gradient" title="Link to this heading">#</a></h2>
<p>In stochastic gradient descent, with and without momentum, we still
have to specify a schedule for tuning the learning rates <span class="math notranslate nohighlight">\(\eta_t\)</span>
as a function of time. As discussed in the context of Newton’s
method, this presents a number of dilemmas. The learning rate is
limited by the steepest direction which can change depending on the
current position in the landscape. To circumvent this problem, ideally
our algorithm would keep track of curvature and take large steps in
shallow, flat directions and small steps in steep, narrow directions.
Second-order methods accomplish this by calculating or approximating
the Hessian and normalizing the learning rate by the
curvature. However, this is very computationally expensive for
extremely large models. Ideally, we would like to be able to
adaptively change the step size to match the landscape without paying
the steep computational price of calculating or approximating
Hessians.</p>
<p>During the last decade a number of methods have been introduced that accomplish
this by tracking not only the gradient, but also the second moment of
the gradient. These methods include AdaGrad, AdaDelta, Root Mean Squared Propagation (RMS-Prop), and
<a class="reference external" href="https://arxiv.org/abs/1412.6980">ADAM</a>.</p>
</section>
<section id="challenge-choosing-a-fixed-learning-rate">
<h2>Challenge: Choosing a Fixed Learning Rate<a class="headerlink" href="#challenge-choosing-a-fixed-learning-rate" title="Link to this heading">#</a></h2>
<p>A fixed <span class="math notranslate nohighlight">\(\eta\)</span> is hard to get right:</p>
<ol class="arabic simple">
<li><p>If <span class="math notranslate nohighlight">\(\eta\)</span> is too large, the updates can overshoot the minimum, causing oscillations or divergence</p></li>
<li><p>If <span class="math notranslate nohighlight">\(\eta\)</span> is too small, convergence is very slow (many iterations to make progress)</p></li>
</ol>
<p>In practice, one often uses trial-and-error or schedules (decaying <span class="math notranslate nohighlight">\(\eta\)</span> over time) to find a workable balance.
For a function with steep directions and flat directions, a single global <span class="math notranslate nohighlight">\(\eta\)</span> may be inappropriate:</p>
<ol class="arabic simple">
<li><p>Steep coordinates require a smaller step size to avoid oscillation.</p></li>
<li><p>Flat/shallow coordinates could use a larger step to speed up progress.</p></li>
<li><p>This issue is pronounced in high-dimensional problems with <strong>sparse or varying-scale features</strong> – we need a method to adjust step sizesper feature.</p></li>
</ol>
</section>
<section id="motivation-for-adaptive-step-sizes">
<h2>Motivation for Adaptive Step Sizes<a class="headerlink" href="#motivation-for-adaptive-step-sizes" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Instead of a fixed global <span class="math notranslate nohighlight">\(\eta\)</span>, use an <strong>adaptive learning rate</strong> for each parameter that depends on the history of gradients.</p></li>
<li><p>Parameters that have large accumulated gradient magnitude should get smaller steps (they’ve been changing a lot), whereas parameters with small or infrequent gradients can have larger relative steps.</p></li>
<li><p>This is especially useful for sparse features: Rarely active features accumulate little gradient, so their learning rate remains comparatively high, ensuring they are not neglected</p></li>
<li><p>Conversely, frequently active features accumulate large gradient sums, and their learning rate automatically decreases, preventing too-large updates</p></li>
<li><p>Several algorithms implement this idea (AdaGrad, RMSProp, AdaDelta, Adam, etc.). We will derive <strong>AdaGrad</strong>, one of the first adaptive methods.</p></li>
</ol>
</section>
<section id="adagrad-algorithm-taken-from-goodfellow-et-al">
<h2>AdaGrad algorithm, taken from <a class="reference external" href="https://www.deeplearningbook.org/contents/optimization.html">Goodfellow et al</a><a class="headerlink" href="#adagrad-algorithm-taken-from-goodfellow-et-al" title="Link to this heading">#</a></h2>
<!-- dom:FIGURE: [figures/adagrad.png, width=600 frac=0.8] -->
<!-- begin figure -->
<p><img src="figures/adagrad.png" width="600"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></section>
<section id="derivation-of-the-adagrad-algorithm">
<h2>Derivation of the AdaGrad Algorithm<a class="headerlink" href="#derivation-of-the-adagrad-algorithm" title="Link to this heading">#</a></h2>
<p><strong>Accumulating Gradient History.</strong></p>
<ol class="arabic simple">
<li><p>AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)</p></li>
<li><p>Let <span class="math notranslate nohighlight">\(g_t = \nabla C_{i_t}(x_t)\)</span> be the gradient at step <span class="math notranslate nohighlight">\(t\)</span> (or a subgradient for nondifferentiable cases).</p></li>
<li><p>Initialize <span class="math notranslate nohighlight">\(r_0 = 0\)</span> (an all-zero vector in <span class="math notranslate nohighlight">\(\mathbb{R}^d\)</span>).</p></li>
<li><p>At each iteration <span class="math notranslate nohighlight">\(t\)</span>, update the accumulation:</p></li>
</ol>
<div class="math notranslate nohighlight">
\[
r_t = r_{t-1} + g_t \circ g_t,
\]</div>
<ol class="arabic simple">
<li><p>Here <span class="math notranslate nohighlight">\(g_t \circ g_t\)</span> denotes element-wise square of the gradient vector. <span class="math notranslate nohighlight">\(g_t^{(j)} = g_{t-1}^{(j)} + (g_{t,j})^2\)</span> for each parameter <span class="math notranslate nohighlight">\(j\)</span>.</p></li>
<li><p>We can view <span class="math notranslate nohighlight">\(H_t = \mathrm{diag}(r_t)\)</span> as a diagonal matrix of past squared gradients. Initially <span class="math notranslate nohighlight">\(H_0 = 0\)</span>.</p></li>
</ol>
</section>
<section id="adagrad-update-rule-derivation">
<h2>AdaGrad Update Rule Derivation<a class="headerlink" href="#adagrad-update-rule-derivation" title="Link to this heading">#</a></h2>
<p>We scale the gradient by the inverse square root of the accumulated matrix <span class="math notranslate nohighlight">\(H_t\)</span>. The AdaGrad update at step <span class="math notranslate nohighlight">\(t\)</span> is:</p>
<div class="math notranslate nohighlight">
\[
\theta_{t+1} =\theta_t - \eta H_t^{-1/2} g_t,
\]</div>
<p>where <span class="math notranslate nohighlight">\(H_t^{-1/2}\)</span> is the diagonal matrix with entries <span class="math notranslate nohighlight">\((r_{t}^{(1)})^{-1/2}, \dots, (r_{t}^{(d)})^{-1/2}\)</span>
In coordinates, this means each parameter <span class="math notranslate nohighlight">\(j\)</span> has an individual step size:</p>
<div class="math notranslate nohighlight">
\[
\theta_{t+1,j} =\theta_{t,j} -\frac{\eta}{\sqrt{r_{t,j}}}g_{t,j}.
\]</div>
<p>In practice we add a small constant <span class="math notranslate nohighlight">\(\epsilon\)</span> in the denominator for numerical stability to avoid division by zero:</p>
<div class="math notranslate nohighlight">
\[
\theta_{t+1,j}= \theta_{t,j}-\frac{\eta}{\sqrt{\epsilon + r_{t,j}}}g_{t,j}.
\]</div>
<p>Equivalently, the effective learning rate for parameter <span class="math notranslate nohighlight">\(j\)</span> at time <span class="math notranslate nohighlight">\(t\)</span> is <span class="math notranslate nohighlight">\(\displaystyle \alpha_{t,j} = \frac{\eta}{\sqrt{\epsilon + r_{t,j}}}\)</span>. This decreases over time as <span class="math notranslate nohighlight">\(r_{t,j}\)</span> grows.</p>
</section>
<section id="adagrad-properties">
<h2>AdaGrad Properties<a class="headerlink" href="#adagrad-properties" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>AdaGrad automatically tunes the step size for each parameter. Parameters with more <em>volatile or large gradients</em> get smaller steps, and those with <em>small or infrequent gradients</em> get relatively larger steps</p></li>
<li><p>No manual schedule needed: The accumulation <span class="math notranslate nohighlight">\(r_t\)</span> keeps increasing (or stays the same if gradient is zero), so step sizes <span class="math notranslate nohighlight">\(\eta/\sqrt{r_t}\)</span> are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.</p></li>
<li><p>Sparse data benefit: For very sparse features, <span class="math notranslate nohighlight">\(r_{t,j}\)</span> grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal</p></li>
<li><p>Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem</p></li>
</ol>
<p>It effectively reduces the need to tune <span class="math notranslate nohighlight">\(\eta\)</span> by hand.</p>
<ol class="arabic simple">
<li><p>Limitations: Because <span class="math notranslate nohighlight">\(r_t\)</span> accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)</p></li>
</ol>
</section>
<section id="rmsprop-adaptive-learning-rates">
<h2>RMSProp: Adaptive Learning Rates<a class="headerlink" href="#rmsprop-adaptive-learning-rates" title="Link to this heading">#</a></h2>
<p>Addresses AdaGrad’s diminishing learning rate issue.
Uses a decaying average of squared gradients (instead of a cumulative sum):</p>
<div class="math notranslate nohighlight">
\[
v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2,
\]</div>
<p>with <span class="math notranslate nohighlight">\(\rho\)</span> typically <span class="math notranslate nohighlight">\(0.9\)</span> (or <span class="math notranslate nohighlight">\(0.99\)</span>).</p>
<ol class="arabic simple">
<li><p>Update: <span class="math notranslate nohighlight">\(\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t)\)</span>.</p></li>
<li><p>Recent gradients have more weight, so <span class="math notranslate nohighlight">\(v_t\)</span> adapts to the current landscape.</p></li>
<li><p>Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.</p></li>
</ol>
<p>RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)</p>
</section>
<section id="rmsprop-algorithm-taken-from-goodfellow-et-al">
<h2>RMSProp algorithm, taken from <a class="reference external" href="https://www.deeplearningbook.org/contents/optimization.html">Goodfellow et al</a><a class="headerlink" href="#rmsprop-algorithm-taken-from-goodfellow-et-al" title="Link to this heading">#</a></h2>
<!-- dom:FIGURE: [figures/rmsprop.png, width=600 frac=0.8] -->
<!-- begin figure -->
<p><img src="figures/rmsprop.png" width="600"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></section>
<section id="adam-optimizer">
<h2>Adam Optimizer<a class="headerlink" href="#adam-optimizer" title="Link to this heading">#</a></h2>
<p>Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.</p>
<ol class="arabic simple">
<li><p>Fast convergence by smoothing gradients (accelerates in long-term gradient direction).</p></li>
<li><p>Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).</p></li>
<li><p>Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)</p></li>
<li><p>Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)</p></li>
</ol>
<p><strong>Result</strong>: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice.</p>
</section>
<section id="id2">
<h2><a class="reference external" href="https://arxiv.org/abs/1412.6980">ADAM optimizer</a><a class="headerlink" href="#id2" title="Link to this heading">#</a></h2>
<p>In <a class="reference external" href="https://arxiv.org/abs/1412.6980">ADAM</a>, we keep a running average of
both the first and second moment of the gradient and use this
information to adaptively change the learning rate for different
parameters. The method is efficient when working with large
problems involving lots data and/or parameters. It is a combination of the
gradient descent with momentum algorithm and the RMSprop algorithm
discussed above.</p>
</section>
<section id="why-combine-momentum-and-rmsprop">
<h2>Why Combine Momentum and RMSProp?<a class="headerlink" href="#why-combine-momentum-and-rmsprop" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).</p></li>
<li><p>Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).</p></li>
<li><p>Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)</p></li>
<li><p>Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)</p></li>
</ol>
<p>Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice</p>
</section>
<section id="adam-exponential-moving-averages-moments">
<h2>Adam: Exponential Moving Averages (Moments)<a class="headerlink" href="#adam-exponential-moving-averages-moments" title="Link to this heading">#</a></h2>
<p>Adam maintains two moving averages at each time step <span class="math notranslate nohighlight">\(t\)</span> for each parameter <span class="math notranslate nohighlight">\(w\)</span>:
<strong>First moment (mean) <span class="math notranslate nohighlight">\(m_t\)</span>.</strong></p>
<p>The Momentum term</p>
<div class="math notranslate nohighlight">
\[
m_t = \beta_1m_{t-1} + (1-\beta_1)\, \nabla C(\theta_t),
\]</div>
<p><strong>Second moment (uncentered variance) <span class="math notranslate nohighlight">\(v_t\)</span>.</strong></p>
<p>The RMS term</p>
<div class="math notranslate nohighlight">
\[
v_t = \beta_2v_{t-1} + (1-\beta_2)(\nabla C(\theta_t))^2,
\]</div>
<p>with typical <span class="math notranslate nohighlight">\(\beta_1 = 0.9\)</span>, <span class="math notranslate nohighlight">\(\beta_2 = 0.999\)</span>. Initialize <span class="math notranslate nohighlight">\(m_0 = 0\)</span>, <span class="math notranslate nohighlight">\(v_0 = 0\)</span>.</p>
<p>These are <strong>biased</strong> estimators of the true first and second moment of the gradients, especially at the start (since <span class="math notranslate nohighlight">\(m_0,v_0\)</span> are zero)</p>
</section>
<section id="adam-bias-correction">
<h2>Adam: Bias Correction<a class="headerlink" href="#adam-bias-correction" title="Link to this heading">#</a></h2>
<p>To counteract initialization bias in <span class="math notranslate nohighlight">\(m_t, v_t\)</span>, Adam computes bias-corrected estimates</p>
<div class="math notranslate nohighlight">
\[
\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}.
\]</div>
<ul class="simple">
<li><p>When <span class="math notranslate nohighlight">\(t\)</span> is small, <span class="math notranslate nohighlight">\(1-\beta_i^t \approx 0\)</span>, so <span class="math notranslate nohighlight">\(\hat{m}_t, \hat{v}_t\)</span> significantly larger than raw <span class="math notranslate nohighlight">\(m_t, v_t\)</span>, compensating for the initial zero bias.</p></li>
<li><p>As <span class="math notranslate nohighlight">\(t\)</span> increases, <span class="math notranslate nohighlight">\(1-\beta_i^t \to 1\)</span>, and <span class="math notranslate nohighlight">\(\hat{m}_t, \hat{v}_t\)</span> converge to <span class="math notranslate nohighlight">\(m_t, v_t\)</span>.</p></li>
<li><p>Bias correction is important for Adam’s stability in early iterations</p></li>
</ul>
</section>
<section id="adam-update-rule-derivation">
<h2>Adam: Update Rule Derivation<a class="headerlink" href="#adam-update-rule-derivation" title="Link to this heading">#</a></h2>
<p>Finally, Adam updates parameters using the bias-corrected moments:</p>
<div class="math notranslate nohighlight">
\[
\theta_{t+1} =\theta_t -\frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon}\hat{m}_t,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\epsilon\)</span> is a small constant (e.g. <span class="math notranslate nohighlight">\(10^{-8}\)</span>) to prevent division by zero.
Breaking it down:</p>
<ol class="arabic simple">
<li><p>Compute gradient <span class="math notranslate nohighlight">\(\nabla C(\theta_t)\)</span>.</p></li>
<li><p>Update first moment <span class="math notranslate nohighlight">\(m_t\)</span> and second moment <span class="math notranslate nohighlight">\(v_t\)</span> (exponential moving averages).</p></li>
<li><p>Bias-correct: <span class="math notranslate nohighlight">\(\hat{m}_t = m_t/(1-\beta_1^t)\)</span>, <span class="math notranslate nohighlight">\(\; \hat{v}_t = v_t/(1-\beta_2^t)\)</span>.</p></li>
<li><p>Compute step: <span class="math notranslate nohighlight">\(\Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}\)</span>.</p></li>
<li><p>Update parameters: <span class="math notranslate nohighlight">\(\theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t\)</span>.</p></li>
</ol>
<p>This is the Adam update rule as given in the original paper.</p>
</section>
<section id="adam-vs-adagrad-and-rmsprop">
<h2>Adam vs. AdaGrad and RMSProp<a class="headerlink" href="#adam-vs-adagrad-and-rmsprop" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)</p></li>
<li><p>RMSProp: Uses moving average of squared gradients (like Adam’s <span class="math notranslate nohighlight">\(v_t\)</span>) to maintain adaptive learning rates, but does not include momentum or bias-correction.</p></li>
<li><p>Adam: Effectively RMSProp + Momentum + Bias-correction</p></li>
</ol>
<ul class="simple">
<li><p>Momentum (<span class="math notranslate nohighlight">\(m_t\)</span>) provides acceleration and smoother convergence.</p></li>
<li><p>Adaptive <span class="math notranslate nohighlight">\(v_t\)</span> scaling moderates the step size per dimension.</p></li>
<li><p>Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.</p></li>
</ul>
<p>In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone</p>
</section>
<section id="adaptivity-across-dimensions">
<h2>Adaptivity Across Dimensions<a class="headerlink" href="#adaptivity-across-dimensions" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.</p></li>
<li><p>This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.</p></li>
<li><p>Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction.</p></li>
</ol>
</section>
<section id="adam-algorithm-taken-from-goodfellow-et-al">
<h2>ADAM algorithm, taken from <a class="reference external" href="https://www.deeplearningbook.org/contents/optimization.html">Goodfellow et al</a><a class="headerlink" href="#adam-algorithm-taken-from-goodfellow-et-al" title="Link to this heading">#</a></h2>
<!-- dom:FIGURE: [figures/adam.png, width=600 frac=0.8] -->
<!-- begin figure -->
<p><img src="figures/adam.png" width="600"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></section>
<section id="algorithms-and-codes-for-adagrad-rmsprop-and-adam">
<h2>Algorithms and codes for Adagrad, RMSprop and Adam<a class="headerlink" href="#algorithms-and-codes-for-adagrad-rmsprop-and-adam" title="Link to this heading">#</a></h2>
<p>The algorithms we have implemented are well described in the text by <a class="reference external" href="https://www.deeplearningbook.org/contents/optimization.html">Goodfellow, Bengio and Courville, chapter 8</a>.</p>
<p>The codes which implement these algorithms are discussed below here.</p>
</section>
<section id="practical-tips">
<h2>Practical tips<a class="headerlink" href="#practical-tips" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p><strong>Randomize the data when making mini-batches</strong>. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.</p></li>
<li><p><strong>Transform your inputs</strong>. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the cost function is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case.</p></li>
<li><p><strong>Monitor the out-of-sample performance.</strong> Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This <em>early stopping</em> significantly improves performance in many settings.</p></li>
<li><p><strong>Adaptive optimization methods don’t always have good generalization.</strong> Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications.</p></li>
</ul>
</section>
<section id="sneaking-in-automatic-differentiation-using-autograd">
<h2>Sneaking in automatic differentiation using Autograd<a class="headerlink" href="#sneaking-in-automatic-differentiation-using-autograd" title="Link to this heading">#</a></h2>
<p>In the examples here we take the liberty of sneaking in automatic
differentiation (without having discussed the mathematics). In
project 1 you will write the gradients as discussed above, that is
hard-coding the gradients. By introducing automatic differentiation
via the library <strong>autograd</strong>, which is now replaced by <strong>JAX</strong>, we have
more flexibility in setting up alternative cost functions.</p>
<p>The
first example shows results with ordinary leats squares.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Autograd to calculate gradients for OLS
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
EigValues, EigVectors = np.linalg.eig(H)
print(f&quot;Eigenvalues of Hessian Matrix:{EigValues}&quot;)
theta = np.random.randn(2,1)
eta = 1.0/np.max(EigValues)
Niterations = 1000
# define the gradient
training_gradient = grad(CostOLS)
for iter in range(Niterations):
gradients = training_gradient(theta)
theta -= eta*gradients
print(&quot;theta from own gd&quot;)
print(theta)
xnew = np.array([[0],[2]])
Xnew = np.c_[np.ones((2,1)), xnew]
ypredict = Xnew.dot(theta)
ypredict2 = Xnew.dot(theta_linreg)
plt.plot(xnew, ypredict, &quot;r-&quot;)
plt.plot(xnew, ypredict2, &quot;b-&quot;)
plt.plot(x, y ,&#39;ro&#39;)
plt.axis([0,2.0,0, 15.0])
plt.xlabel(r&#39;$x$&#39;)
plt.ylabel(r&#39;$y$&#39;)
plt.title(r&#39;Random numbers &#39;)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="id3">
<h2>Same code but now with momentum gradient descent<a class="headerlink" href="#id3" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Autograd to calculate gradients for OLS
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x#+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
EigValues, EigVectors = np.linalg.eig(H)
print(f&quot;Eigenvalues of Hessian Matrix:{EigValues}&quot;)
theta = np.random.randn(2,1)
eta = 1.0/np.max(EigValues)
Niterations = 30
# define the gradient
training_gradient = grad(CostOLS)
for iter in range(Niterations):
gradients = training_gradient(theta)
theta -= eta*gradients
print(iter,gradients[0],gradients[1])
print(&quot;theta from own gd&quot;)
print(theta)
# Now improve with momentum gradient descent
change = 0.0
delta_momentum = 0.3
for iter in range(Niterations):
# calculate gradient
gradients = training_gradient(theta)
# calculate update
new_change = eta*gradients+delta_momentum*change
# take a step
theta -= new_change
# save the change
change = new_change
print(iter,gradients[0],gradients[1])
print(&quot;theta from own gd wth momentum&quot;)
print(theta)
</pre></div>
</div>
</div>
</div>
</section>
<section id="including-stochastic-gradient-descent-with-autograd">
<h2>Including Stochastic Gradient Descent with Autograd<a class="headerlink" href="#including-stochastic-gradient-descent-with-autograd" title="Link to this heading">#</a></h2>
<p>In this code we include the stochastic gradient descent approach
discussed above. Note here that we specify which argument we are
taking the derivative with respect to when using <strong>autograd</strong>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Autograd to calculate gradients using SGD
# OLS example
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
# Note change from previous example
def CostOLS(y,X,theta):
return np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
EigValues, EigVectors = np.linalg.eig(H)
print(f&quot;Eigenvalues of Hessian Matrix:{EigValues}&quot;)
theta = np.random.randn(2,1)
eta = 1.0/np.max(EigValues)
Niterations = 1000
# Note that we request the derivative wrt third argument (theta, 2 here)
training_gradient = grad(CostOLS,2)
for iter in range(Niterations):
gradients = (1.0/n)*training_gradient(y, X, theta)
theta -= eta*gradients
print(&quot;theta from own gd&quot;)
print(theta)
xnew = np.array([[0],[2]])
Xnew = np.c_[np.ones((2,1)), xnew]
ypredict = Xnew.dot(theta)
ypredict2 = Xnew.dot(theta_linreg)
plt.plot(xnew, ypredict, &quot;r-&quot;)
plt.plot(xnew, ypredict2, &quot;b-&quot;)
plt.plot(x, y ,&#39;ro&#39;)
plt.axis([0,2.0,0, 15.0])
plt.xlabel(r&#39;$x$&#39;)
plt.ylabel(r&#39;$y$&#39;)
plt.title(r&#39;Random numbers &#39;)
plt.show()
n_epochs = 50
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
t0, t1 = 5, 50
def learning_schedule(t):
return t0/(t+t1)
theta = np.random.randn(2,1)
for epoch in range(n_epochs):
# Can you figure out a better way of setting up the contributions to each batch?
for i in range(m):
random_index = M*np.random.randint(m)
xi = X[random_index:random_index+M]
yi = y[random_index:random_index+M]
gradients = (1.0/M)*training_gradient(yi, xi, theta)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
print(&quot;theta from own sdg&quot;)
print(theta)
</pre></div>
</div>
</div>
</div>
</section>
<section id="id4">
<h2>Same code but now with momentum gradient descent<a class="headerlink" href="#id4" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Autograd to calculate gradients using SGD
# OLS example
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
# Note change from previous example
def CostOLS(y,X,theta):
return np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
EigValues, EigVectors = np.linalg.eig(H)
print(f&quot;Eigenvalues of Hessian Matrix:{EigValues}&quot;)
theta = np.random.randn(2,1)
eta = 1.0/np.max(EigValues)
Niterations = 100
# Note that we request the derivative wrt third argument (theta, 2 here)
training_gradient = grad(CostOLS,2)
for iter in range(Niterations):
gradients = (1.0/n)*training_gradient(y, X, theta)
theta -= eta*gradients
print(&quot;theta from own gd&quot;)
print(theta)
n_epochs = 50
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
t0, t1 = 5, 50
def learning_schedule(t):
return t0/(t+t1)
theta = np.random.randn(2,1)
change = 0.0
delta_momentum = 0.3
for epoch in range(n_epochs):
for i in range(m):
random_index = M*np.random.randint(m)
xi = X[random_index:random_index+M]
yi = y[random_index:random_index+M]
gradients = (1.0/M)*training_gradient(yi, xi, theta)
eta = learning_schedule(epoch*m+i)
# calculate update
new_change = eta*gradients+delta_momentum*change
# take a step
theta -= new_change
# save the change
change = new_change
print(&quot;theta from own sdg with momentum&quot;)
print(theta)
</pre></div>
</div>
</div>
</div>
</section>
<section id="but-none-of-these-can-compete-with-newton-s-method">
<h2>But none of these can compete with Newton’s method<a class="headerlink" href="#but-none-of-these-can-compete-with-newton-s-method" title="Link to this heading">#</a></h2>
<p>Note that we here have introduced automatic differentiation</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Newton&#39;s method
from random import random, seed
import numpy as np
import autograd.numpy as np
from autograd import grad
def CostOLS(theta):
return (1.0/n)*np.sum((y-X @ theta)**2)
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+5*x*x
X = np.c_[np.ones((n,1)), x, x*x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Hessian matrix
H = (2.0/n)* XT_X
# Note that here the Hessian does not depend on the parameters theta
invH = np.linalg.pinv(H)
theta = np.random.randn(3,1)
Niterations = 5
# define the gradient
training_gradient = grad(CostOLS)
for iter in range(Niterations):
gradients = training_gradient(theta)
theta -= invH @ gradients
print(iter,gradients[0],gradients[1])
print(&quot;theta from own Newton code&quot;)
print(theta)
</pre></div>
</div>
</div>
</div>
</section>
<section id="similar-second-order-function-now-problem-but-now-with-adagrad">
<h2>Similar (second order function now) problem but now with AdaGrad<a class="headerlink" href="#similar-second-order-function-now-problem-but-now-with-adagrad" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent
# OLS example
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
# Note change from previous example
def CostOLS(y,X,theta):
return np.sum((y-X @ theta)**2)
n = 1000
x = np.random.rand(n,1)
y = 2.0+3*x +4*x*x
X = np.c_[np.ones((n,1)), x, x*x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Note that we request the derivative wrt third argument (theta, 2 here)
training_gradient = grad(CostOLS,2)
# Define parameters for Stochastic Gradient Descent
n_epochs = 50
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
# Guess for unknown parameters theta
theta = np.random.randn(3,1)
# Value for learning rate
eta = 0.01
# Including AdaGrad parameter to avoid possible division by zero
delta = 1e-8
for epoch in range(n_epochs):
Giter = 0.0
for i in range(m):
random_index = M*np.random.randint(m)
xi = X[random_index:random_index+M]
yi = y[random_index:random_index+M]
gradients = (1.0/M)*training_gradient(yi, xi, theta)
Giter += gradients*gradients
update = gradients*eta/(delta+np.sqrt(Giter))
theta -= update
print(&quot;theta from own AdaGrad&quot;)
print(theta)
</pre></div>
</div>
</div>
</div>
<p>Running this code we note an almost perfect agreement with the results from matrix inversion.</p>
</section>
<section id="rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent">
<h2>RMSprop for adaptive learning rate with Stochastic Gradient Descent<a class="headerlink" href="#rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent
# OLS example
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
# Note change from previous example
def CostOLS(y,X,theta):
return np.sum((y-X @ theta)**2)
n = 1000
x = np.random.rand(n,1)
y = 2.0+3*x +4*x*x# +np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x, x*x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Note that we request the derivative wrt third argument (theta, 2 here)
training_gradient = grad(CostOLS,2)
# Define parameters for Stochastic Gradient Descent
n_epochs = 50
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
# Guess for unknown parameters theta
theta = np.random.randn(3,1)
# Value for learning rate
eta = 0.01
# Value for parameter rho
rho = 0.99
# Including AdaGrad parameter to avoid possible division by zero
delta = 1e-8
for epoch in range(n_epochs):
Giter = 0.0
for i in range(m):
random_index = M*np.random.randint(m)
xi = X[random_index:random_index+M]
yi = y[random_index:random_index+M]
gradients = (1.0/M)*training_gradient(yi, xi, theta)
# Accumulated gradient
# Scaling with rho the new and the previous results
Giter = (rho*Giter+(1-rho)*gradients*gradients)
# Taking the diagonal only and inverting
update = gradients*eta/(delta+np.sqrt(Giter))
# Hadamard product
theta -= update
print(&quot;theta from own RMSprop&quot;)
print(theta)
</pre></div>
</div>
</div>
</div>
</section>
<section id="and-finally-adam">
<h2>And finally <a class="reference external" href="https://arxiv.org/pdf/1412.6980.pdf">ADAM</a><a class="headerlink" href="#and-finally-adam" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent
# OLS example
from random import random, seed
import numpy as np
import autograd.numpy as np
import matplotlib.pyplot as plt
from autograd import grad
# Note change from previous example
def CostOLS(y,X,theta):
return np.sum((y-X @ theta)**2)
n = 1000
x = np.random.rand(n,1)
y = 2.0+3*x +4*x*x# +np.random.randn(n,1)
X = np.c_[np.ones((n,1)), x, x*x]
XT_X = X.T @ X
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
print(&quot;Own inversion&quot;)
print(theta_linreg)
# Note that we request the derivative wrt third argument (theta, 2 here)
training_gradient = grad(CostOLS,2)
# Define parameters for Stochastic Gradient Descent
n_epochs = 50
M = 5 #size of each minibatch
m = int(n/M) #number of minibatches
# Guess for unknown parameters theta
theta = np.random.randn(3,1)
# Value for learning rate
eta = 0.01
# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980
theta1 = 0.9
theta2 = 0.999
# Including AdaGrad parameter to avoid possible division by zero
delta = 1e-7
iter = 0
for epoch in range(n_epochs):
first_moment = 0.0
second_moment = 0.0
iter += 1
for i in range(m):
random_index = M*np.random.randint(m)
xi = X[random_index:random_index+M]
yi = y[random_index:random_index+M]
gradients = (1.0/M)*training_gradient(yi, xi, theta)
# Computing moments first
first_moment = theta1*first_moment + (1-theta1)*gradients
second_moment = theta2*second_moment+(1-theta2)*gradients*gradients
first_term = first_moment/(1.0-theta1**iter)
second_term = second_moment/(1.0-theta2**iter)
# Scaling with rho the new and the previous results
update = eta*first_term/(np.sqrt(second_term)+delta)
theta -= update
print(&quot;theta from own ADAM&quot;)
print(theta)
</pre></div>
</div>
</div>
</div>
</section>
<section id="material-for-the-lab-sessions">
<h2>Material for the lab sessions<a class="headerlink" href="#material-for-the-lab-sessions" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)</p></li>
<li><p>Work on project 1</p></li>
</ol>
<!-- * [Video of exercise sessions week 37](https://youtu.be/bK4AEcTu-oM) -->
<p>For more discussions of Ridge regression and calculation of averages, <a class="reference external" href="https://arxiv.org/abs/1509.09169">Wessel van Wieringen’s</a> article is highly recommended.</p>
</section>
<section id="reminder-on-different-scaling-methods">
<h2>Reminder on different scaling methods<a class="headerlink" href="#reminder-on-different-scaling-methods" title="Link to this heading">#</a></h2>
<p>Before fitting a regression model, it is good practice to normalize or
standardize the features. This ensures all features are on a
comparable scale, which is especially important when using
regularization. In the exercises this week we will perform standardization, scaling each
feature to have mean 0 and standard deviation 1.</p>
<p>Here we compute the mean and standard deviation of each column (feature) in our design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.
Then we subtract the mean and divide by the standard deviation for each feature.</p>
<p>In the example here we
we will also center the target <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> to mean <span class="math notranslate nohighlight">\(0\)</span>. Centering <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>
(and each feature) means the model does not require a separate intercept
term, the data is shifted such that the intercept is effectively 0
. (In practice, one could include an intercept in the model and not
penalize it, but here we simplify by centering.)
Choose <span class="math notranslate nohighlight">\(n=100\)</span> data points and set up <span class="math notranslate nohighlight">\(\boldsymbol{x}, \)</span>\boldsymbol{y}<span class="math notranslate nohighlight">\( and the design matrix \)</span>\boldsymbol{X}$.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Standardize features (zero mean, unit variance for each feature)
X_mean = X.mean(axis=0)
X_std = X.std(axis=0)
X_std[X_std == 0] = 1 # safeguard to avoid division by zero for constant features
X_norm = (X - X_mean) / X_std
# Center the target to zero mean (optional, to simplify intercept handling)
y_mean = ?
y_centered = ?
</pre></div>
</div>
</div>
</div>
<p>Do we need to center the values of <span class="math notranslate nohighlight">\(y\)</span>?</p>
<p>After this preprocessing, each column of <span class="math notranslate nohighlight">\(\boldsymbol{X}_{\mathrm{norm}}\)</span> has mean zero and standard deviation <span class="math notranslate nohighlight">\(1\)</span>
and <span class="math notranslate nohighlight">\(\boldsymbol{y}_{\mathrm{centered}}\)</span> has mean 0. This can make the optimization landscape
nicer and ensures the regularization penalty <span class="math notranslate nohighlight">\(\lambda \sum_j
\theta_j^2\)</span> in Ridge regression treats each coefficient fairly (since features are on the
same scale).</p>
</section>
<section id="functionality-in-scikit-learn">
<h2>Functionality in Scikit-Learn<a class="headerlink" href="#functionality-in-scikit-learn" title="Link to this heading">#</a></h2>
<p><strong>Scikit-Learn</strong> has several functions which allow us to rescale the
data, normally resulting in much better results in terms of various
accuracy scores. The <strong>StandardScaler</strong> function in <strong>Scikit-Learn</strong>
ensures that for each feature/predictor we study the mean value is
zero and the variance is one (every column in the design/feature
matrix). This scaling has the drawback that it does not ensure that
we have a particular maximum or minimum in our data set. Another
function included in <strong>Scikit-Learn</strong> is the <strong>MinMaxScaler</strong> which
ensures that all features are exactly between <span class="math notranslate nohighlight">\(0\)</span> and <span class="math notranslate nohighlight">\(1\)</span>. The</p>
</section>
<section id="more-preprocessing">
<h2>More preprocessing<a class="headerlink" href="#more-preprocessing" title="Link to this heading">#</a></h2>
<p>The <strong>Normalizer</strong> scales each data
point such that the feature vector has a euclidean length of one. In other words, it
projects a data point on the circle (or sphere in the case of higher dimensions) with a
radius of 1. This means every data point is scaled by a different number (by the
inverse of it’s length).
This normalization is often used when only the direction (or angle) of the data matters,
not the length of the feature vector.</p>
<p>The <strong>RobustScaler</strong> works similarly to the StandardScaler in that it
ensures statistical properties for each feature that guarantee that
they are on the same scale. However, the RobustScaler uses the median
and quartiles, instead of mean and variance. This makes the
RobustScaler ignore data points that are very different from the rest
(like measurement errors). These odd data points are also called
outliers, and might often lead to trouble for other scaling
techniques.</p>
</section>
<section id="frequently-used-scaling-functions">
<h2>Frequently used scaling functions<a class="headerlink" href="#frequently-used-scaling-functions" title="Link to this heading">#</a></h2>
<p>Many features are often scaled using standardization to improve performance. In <strong>Scikit-Learn</strong> this is given by the <strong>StandardScaler</strong> function as discussed above. It is easy however to write your own.
Mathematically, this involves subtracting the mean and divide by the standard deviation over the data set, for each feature:</p>
<div class="math notranslate nohighlight">
\[
x_j^{(i)} \rightarrow \frac{x_j^{(i)} - \overline{x}_j}{\sigma(x_j)},
\]</div>
<p>where <span class="math notranslate nohighlight">\(\overline{x}_j\)</span> and <span class="math notranslate nohighlight">\(\sigma(x_j)\)</span> are the mean and standard deviation, respectively, of the feature <span class="math notranslate nohighlight">\(x_j\)</span>.
This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don’t wish to calculate it, it is then common to simply set it to one.</p>
<p>Keep in mind that when you transform your data set before training a model, the same transformation needs to be done
on your eventual new data set before making a prediction. If we translate this into a Python code, it would could be implemented as</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>&quot;&quot;&quot;
#Model training, we compute the mean value of y and X
y_train_mean = np.mean(y_train)
X_train_mean = np.mean(X_train,axis=0)
X_train = X_train - X_train_mean
y_train = y_train - y_train_mean
# The we fit our model with the training data
trained_model = some_model.fit(X_train,y_train)
#Model prediction, we need also to transform our data set used for the prediction.
X_test = X_test - X_train_mean #Use mean from training data
y_pred = trained_model(X_test)
y_pred = y_pred + y_train_mean
&quot;&quot;&quot;
</pre></div>
</div>
</div>
</div>
<p>Let us try to understand what this may imply mathematically when we
subtract the mean values, also known as <em>zero centering</em>. For
simplicity, we will focus on ordinary regression, as done in the above example.</p>
<p>The cost/loss function for regression is</p>
<div class="math notranslate nohighlight">
\[
C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,.
\]</div>
<p>Recall also that we use the squared value. This expression can lead to an
increased penalty for higher differences between predicted and
output/target values.</p>
<p>What we have done is to single out the <span class="math notranslate nohighlight">\(\theta_0\)</span> term in the
definition of the mean squared error (MSE). The design matrix <span class="math notranslate nohighlight">\(X\)</span>
does in this case not contain any intercept column. When we take the
derivative with respect to <span class="math notranslate nohighlight">\(\theta_0\)</span>, we want the derivative to obey</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C}{\partial \theta_j} = 0,
\]</div>
<p>for all <span class="math notranslate nohighlight">\(j\)</span>. For <span class="math notranslate nohighlight">\(\theta_0\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right).
\]</div>
<p>Multiplying away the constant <span class="math notranslate nohighlight">\(2/n\)</span>, we obtain</p>
<div class="math notranslate nohighlight">
\[
\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j.
\]</div>
<p>Let us specialize first to the case where we have only two parameters <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>.
Our result for <span class="math notranslate nohighlight">\(\theta_0\)</span> simplifies then to</p>
<div class="math notranslate nohighlight">
\[
n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1.
\]</div>
<p>We obtain then</p>
<div class="math notranslate nohighlight">
\[
\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
\]</div>
<p>If we define</p>
<div class="math notranslate nohighlight">
\[
\mu_{\boldsymbol{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
\]</div>
<p>and the mean value of the outputs as</p>
<div class="math notranslate nohighlight">
\[
\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
\]</div>
<p>we have</p>
<div class="math notranslate nohighlight">
\[
\theta_0 = \mu_y - \theta_1\mu_{\boldsymbol{x}_1}.
\]</div>
<p>In the general case with more parameters than <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j.
\]</div>
<p>We can rewrite the latter equation as</p>
<div class="math notranslate nohighlight">
\[
\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\theta_j,
\]</div>
<p>where we have defined</p>
<div class="math notranslate nohighlight">
\[
\mu_{\boldsymbol{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
\]</div>
<p>the mean value for all elements of the column vector <span class="math notranslate nohighlight">\(\boldsymbol{x}_j\)</span>.</p>
<p>Replacing <span class="math notranslate nohighlight">\(y_i\)</span> with <span class="math notranslate nohighlight">\(y_i - y_i - \overline{\boldsymbol{y}}\)</span> and centering also our design matrix results in a cost function (in vector-matrix disguise)</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}).
\]</div>
<p>If we minimize with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we have then</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}}\)</span>
and <span class="math notranslate nohighlight">\(\tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}\)</span>.</p>
<p>For Ridge regression we need to add <span class="math notranslate nohighlight">\(\lambda \boldsymbol{\theta}^T\boldsymbol{\theta}\)</span> to the cost function and get then</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
\]</div>
<p>What does this mean? And why do we insist on all this? Let us look at some examples.</p>
<p>This code shows a simple first-order fit to a data set using the above transformed data, where we consider the role of the intercept first, by either excluding it or including it (<em>code example thanks to Øyvind Sigmundson Schøyen</em>). Here our scaling of the data is done by subtracting the mean values only.
Note also that we do not split the data into training and test.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
np.random.seed(2021)
def MSE(y_data,y_model):
n = np.size(y_model)
return np.sum((y_data-y_model)**2)/n
def fit_theta(X, y):
return np.linalg.pinv(X.T @ X) @ X.T @ y
true_theta = [2, 0.5, 3.7]
x = np.linspace(0, 1, 11)
y = np.sum(
np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0
) + 0.1 * np.random.normal(size=len(x))
degree = 3
X = np.zeros((len(x), degree))
# Include the intercept in the design matrix
for p in range(degree):
X[:, p] = x ** p
theta = fit_theta(X, y)
# Intercept is included in the design matrix
skl = LinearRegression(fit_intercept=False).fit(X, y)
print(f&quot;True theta: {true_theta}&quot;)
print(f&quot;Fitted theta: {theta}&quot;)
print(f&quot;Sklearn fitted theta: {skl.coef_}&quot;)
ypredictOwn = X @ theta
ypredictSKL = skl.predict(X)
print(f&quot;MSE with intercept column&quot;)
print(MSE(y,ypredictOwn))
print(f&quot;MSE with intercept column from SKL&quot;)
print(MSE(y,ypredictSKL))
plt.figure()
plt.scatter(x, y, label=&quot;Data&quot;)
plt.plot(x, X @ theta, label=&quot;Fit&quot;)
plt.plot(x, skl.predict(X), label=&quot;Sklearn (fit_intercept=False)&quot;)
# Do not include the intercept in the design matrix
X = np.zeros((len(x), degree - 1))
for p in range(degree - 1):
X[:, p] = x ** (p + 1)
# Intercept is not included in the design matrix
skl = LinearRegression(fit_intercept=True).fit(X, y)
# Use centered values for X and y when computing coefficients
y_offset = np.average(y, axis=0)
X_offset = np.average(X, axis=0)
theta = fit_theta(X - X_offset, y - y_offset)
intercept = np.mean(y_offset - X_offset @ theta)
print(f&quot;Manual intercept: {intercept}&quot;)
print(f&quot;Fitted theta (without intercept): {theta}&quot;)
print(f&quot;Sklearn intercept: {skl.intercept_}&quot;)
print(f&quot;Sklearn fitted theta (without intercept): {skl.coef_}&quot;)
ypredictOwn = X @ theta
ypredictSKL = skl.predict(X)
print(f&quot;MSE with Manual intercept&quot;)
print(MSE(y,ypredictOwn+intercept))
print(f&quot;MSE with Sklearn intercept&quot;)
print(MSE(y,ypredictSKL))
plt.plot(x, X @ theta + intercept, &quot;--&quot;, label=&quot;Fit (manual intercept)&quot;)
plt.plot(x, skl.predict(X), &quot;--&quot;, label=&quot;Sklearn (fit_intercept=True)&quot;)
plt.grid()
plt.legend()
plt.show()
</pre></div>
</div>
</div>
</div>
<p>The intercept is the value of our output/target variable
when all our features are zero and our function crosses the <span class="math notranslate nohighlight">\(y\)</span>-axis (for a one-dimensional case).</p>
<p>Printing the MSE, we see first that both methods give the same MSE, as
they should. However, when we move to for example Ridge regression,
the way we treat the intercept may give a larger or smaller MSE,
meaning that the MSE can be penalized by the value of the
intercept. Not including the intercept in the fit, means that the
regularization term does not include <span class="math notranslate nohighlight">\(\theta_0\)</span>. For different values
of <span class="math notranslate nohighlight">\(\lambda\)</span>, this may lead to different MSE values.</p>
<p>To remind the reader, the regularization term, with the intercept in Ridge regression, is given by</p>
<div class="math notranslate nohighlight">
\[
\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2,
\]</div>
<p>but when we take out the intercept, this equation becomes</p>
<div class="math notranslate nohighlight">
\[
\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2.
\]</div>
<p>For Lasso regression we have</p>
<div class="math notranslate nohighlight">
\[
\lambda \vert\vert \boldsymbol{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert.
\]</div>
<p>It means that, when scaling the design matrix and the outputs/targets,
by subtracting the mean values, we have an optimization problem which
is not penalized by the intercept. The MSE value can then be smaller
since it focuses only on the remaining quantities. If we however bring
back the intercept, we will get a MSE which then contains the
intercept.</p>
<p>Armed with this wisdom, we attempt first to simply set the intercept equal to <strong>False</strong> in our implementation of Ridge regression for our well-known vanilla data set.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn import linear_model
def MSE(y_data,y_model):
n = np.size(y_model)
return np.sum((y_data-y_model)**2)/n
# A seed just to ensure that the random numbers are the same for every run.
# Useful for eventual debugging.
np.random.seed(3155)
n = 100
x = np.random.rand(n)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
Maxpolydegree = 20
X = np.zeros((n,Maxpolydegree))
#We include explicitely the intercept column
for degree in range(Maxpolydegree):
X[:,degree] = x**degree
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
p = Maxpolydegree
I = np.eye(p,p)
# Decide which values of lambda to use
nlambdas = 6
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
# Note: we include the intercept column and no scaling
RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
RegRidge.fit(X_train,y_train)
# and then make the prediction
ytildeOwnRidge = X_train @ OwnRidgeTheta
ypredictOwnRidge = X_test @ OwnRidgeTheta
ytildeRidge = RegRidge.predict(X_train)
ypredictRidge = RegRidge.predict(X_test)
MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
MSERidgePredict[i] = MSE(y_test,ypredictRidge)
print(&quot;Theta values for own Ridge implementation&quot;)
print(OwnRidgeTheta)
print(&quot;Theta values for Scikit-Learn Ridge implementation&quot;)
print(RegRidge.coef_)
print(&quot;MSE values for own Ridge implementation&quot;)
print(MSEOwnRidgePredict[i])
print(&quot;MSE values for Scikit-Learn Ridge implementation&quot;)
print(MSERidgePredict[i])
# Now plot the results
plt.figure()
plt.plot(np.log10(lambdas), MSEOwnRidgePredict, &#39;r&#39;, label = &#39;MSE own Ridge Test&#39;)
plt.plot(np.log10(lambdas), MSERidgePredict, &#39;g&#39;, label = &#39;MSE Ridge Test&#39;)
plt.xlabel(&#39;log10(lambda)&#39;)
plt.ylabel(&#39;MSE&#39;)
plt.legend()
plt.show()
</pre></div>
</div>
</div>
</div>
<p>The results here agree when we force <strong>Scikit-Learn</strong>’s Ridge function to include the first column in our design matrix.
We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.
What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn import linear_model
from sklearn.preprocessing import StandardScaler
def MSE(y_data,y_model):
n = np.size(y_model)
return np.sum((y_data-y_model)**2)/n
# A seed just to ensure that the random numbers are the same for every run.
# Useful for eventual debugging.
np.random.seed(315)
n = 100
x = np.random.rand(n)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
Maxpolydegree = 20
X = np.zeros((n,Maxpolydegree-1))
for degree in range(1,Maxpolydegree): #No intercept column
X[:,degree-1] = x**(degree)
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable
X_train_mean = np.mean(X_train,axis=0)
#Center by removing mean from each feature
X_train_scaled = X_train - X_train_mean
X_test_scaled = X_test - X_train_mean
#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)
#Remove the intercept from the training data.
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
p = Maxpolydegree-1
I = np.eye(p,p)
# Decide which values of lambda to use
nlambdas = 6
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data
#Add intercept to prediction
ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler
RegRidge = linear_model.Ridge(lmb)
RegRidge.fit(X_train,y_train)
ypredictRidge = RegRidge.predict(X_test)
MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
MSERidgePredict[i] = MSE(y_test,ypredictRidge)
print(&quot;Theta values for own Ridge implementation&quot;)
print(OwnRidgeTheta) #Intercept is given by mean of target variable
print(&quot;Theta values for Scikit-Learn Ridge implementation&quot;)
print(RegRidge.coef_)
print(&#39;Intercept from own implementation:&#39;)
print(intercept_)
print(&#39;Intercept from Scikit-Learn Ridge implementation&#39;)
print(RegRidge.intercept_)
print(&quot;MSE values for own Ridge implementation&quot;)
print(MSEOwnRidgePredict[i])
print(&quot;MSE values for Scikit-Learn Ridge implementation&quot;)
print(MSERidgePredict[i])
# Now plot the results
plt.figure()
plt.plot(np.log10(lambdas), MSEOwnRidgePredict, &#39;b--&#39;, label = &#39;MSE own Ridge Test&#39;)
plt.plot(np.log10(lambdas), MSERidgePredict, &#39;g--&#39;, label = &#39;MSE SL Ridge Test&#39;)
plt.xlabel(&#39;log10(lambda)&#39;)
plt.ylabel(&#39;MSE&#39;)
plt.legend()
plt.show()
</pre></div>
</div>
</div>
</div>
<p>We see here, when compared to the code which includes explicitely the
intercept column, that our MSE value is actually smaller. This is
because the regularization term does not include the intercept value
<span class="math notranslate nohighlight">\(\theta_0\)</span> in the fitting. This applies to Lasso regularization as
well. It means that our optimization is now done only with the
centered matrix and/or vector that enter the fitting procedure.</p>
</section>
</section>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
name: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</article>
<footer class="prev-next-footer d-print-none">
<div class="prev-next-area">
<a class="left-prev"
href="exercisesweek37.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Exercises week 37</p>
</div>
</a>
<a class="right-next"
href="exercisesweek38.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Exercises week 38</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
</div>
</footer>
</div>
<div class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
<div class="sidebar-secondary-item">
<div class="page-toc tocsection onthispage">
<i class="fa-solid fa-list"></i> Contents
</div>
<nav class="bd-toc-nav page-toc">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#plans-for-week-37-lecture-monday">Plans for week 37, lecture Monday</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#readings-and-videos">Readings and Videos:</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#material-for-lecture-monday-september-8">Material for lecture Monday September 8</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-and-revisiting-ordinary-least-squares-from-last-week">Gradient descent and revisiting Ordinary Least Squares from last week</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-example">Gradient descent example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-derivative-of-the-cost-loss-function">The derivative of the cost/loss function</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix">The Hessian matrix</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-program">Simple program</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Gradient Descent Example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-and-ridge">Gradient descent and Ridge</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix-for-ridge-regression">The Hessian matrix for Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#using-gradient-descent-methods-limitations">Using gradient descent methods, limitations</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#momentum-based-gd">Momentum based GD</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#improving-gradient-descent-with-momentum">Improving gradient descent with momentum</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#same-code-but-now-with-momentum-gradient-descent">Same code but now with momentum gradient descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#overview-video-on-stochastic-gradient-descent-sgd">Overview video on Stochastic Gradient Descent (SGD)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#batches-and-mini-batches">Batches and mini-batches</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#pros-and-cons">Pros and cons</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#convergence-rates">Convergence rates</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#accuracy">Accuracy</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#stochastic-gradient-descent">Stochastic Gradient Descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#computation-of-gradients">Computation of gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sgd-example">SGD example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-gradient-step">The gradient step</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-example-code">Simple example code</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#when-do-we-stop">When do we stop?</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#slightly-different-approach">Slightly different approach</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#time-decay-rate">Time decay rate</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#code-with-a-number-of-minibatches-which-varies">Code with a Number of Minibatches which varies</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#replace-or-not">Replace or not</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison">SGD vs Full-Batch GD: Convergence Speed and Memory Comparison</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#theoretical-convergence-speed-and-convex-optimization">Theoretical Convergence Speed and convex optimization</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#strongly-convex-case">Strongly Convex Case</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#non-convex-problems">Non-Convex Problems</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-usage-and-scalability">Memory Usage and Scalability</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#empirical-evidence-convergence-time-and-memory-in-practice">Empirical Evidence: Convergence Time and Memory in Practice</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deep-neural-networks">Deep Neural Networks</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-constraints">Memory constraints</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#second-moment-of-the-gradient">Second moment of the gradient</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#challenge-choosing-a-fixed-learning-rate">Challenge: Choosing a Fixed Learning Rate</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#motivation-for-adaptive-step-sizes">Motivation for Adaptive Step Sizes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adagrad-algorithm-taken-from-goodfellow-et-al">AdaGrad algorithm, taken from Goodfellow et al</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#derivation-of-the-adagrad-algorithm">Derivation of the AdaGrad Algorithm</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adagrad-update-rule-derivation">AdaGrad Update Rule Derivation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adagrad-properties">AdaGrad Properties</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rmsprop-adaptive-learning-rates">RMSProp: Adaptive Learning Rates</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rmsprop-algorithm-taken-from-goodfellow-et-al">RMSProp algorithm, taken from Goodfellow et al</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-optimizer">Adam Optimizer</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id2">ADAM optimizer</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-combine-momentum-and-rmsprop">Why Combine Momentum and RMSProp?</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-exponential-moving-averages-moments">Adam: Exponential Moving Averages (Moments)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-bias-correction">Adam: Bias Correction</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-update-rule-derivation">Adam: Update Rule Derivation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-vs-adagrad-and-rmsprop">Adam vs. AdaGrad and RMSProp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adaptivity-across-dimensions">Adaptivity Across Dimensions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adam-algorithm-taken-from-goodfellow-et-al">ADAM algorithm, taken from Goodfellow et al</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#algorithms-and-codes-for-adagrad-rmsprop-and-adam">Algorithms and codes for Adagrad, RMSprop and Adam</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#practical-tips">Practical tips</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sneaking-in-automatic-differentiation-using-autograd">Sneaking in automatic differentiation using Autograd</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id3">Same code but now with momentum gradient descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">Including Stochastic Gradient Descent with Autograd</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id4">Same code but now with momentum gradient descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#but-none-of-these-can-compete-with-newton-s-method">But none of these can compete with Newton’s method</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#similar-second-order-function-now-problem-but-now-with-adagrad">Similar (second order function now) problem but now with AdaGrad</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent">RMSprop for adaptive learning rate with Stochastic Gradient Descent</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#and-finally-adam">And finally ADAM</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#material-for-the-lab-sessions">Material for the lab sessions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reminder-on-different-scaling-methods">Reminder on different scaling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#functionality-in-scikit-learn">Functionality in Scikit-Learn</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#more-preprocessing">More preprocessing</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#frequently-used-scaling-functions">Frequently used scaling functions</a></li>
</ul>
</nav></div>
</div></div>
</div>
<footer class="bd-footer-content">
<div class="bd-footer-content__inner container">
<div class="footer-item">
<p class="component-author">
By Morten Hjorth-Jensen
</p>
</div>
<div class="footer-item">
<p class="copyright">
© Copyright 2023.
<br/>
</p>
</div>
<div class="footer-item">
</div>
<div class="footer-item">
</div>
</div>
</footer>
</main>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script src="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script>
<footer class="bd-footer">
</footer>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink