diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index be839261c..41110a69c 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -421,7 +421,7 @@ MathJax.Hub.Config({
-

Nov 1, 2022

+

Week 39


@@ -460,7 +460,7 @@ MathJax.Hub.Config({ -->
- © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index b2dd48fc5..a399dfa6a 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -184,13 +184,13 @@ MathJax.Hub.Config({
-

Nov 1, 2022

+

Week 39


- © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
@@ -198,18 +198,8 @@ MathJax.Hub.Config({

Plan for week 39

diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index 5bda3166c..b33720b01 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -335,7 +335,7 @@ MathJax.Hub.Config({

-

Nov 1, 2022

+

Week 39


@@ -343,14 +343,8 @@ MathJax.Hub.Config({

Plan for week 39

See lecture notes for week 39. @@ -3785,7 +3779,7 @@ derivative_fn = grad(sum_logistic)

- © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index d3345667e..74ad42395 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -412,7 +412,7 @@ MathJax.Hub.Config({
-

Nov 1, 2022

+

Week 39


@@ -420,14 +420,8 @@ MathJax.Hub.Config({

Plan for week 39

See lecture notes for week 39. @@ -3862,7 +3856,7 @@ derivative_fn = grad(sum_logistic)

- © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index f3e44c597..e9ca1fcbc 100644 Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 9cbd5e3ac..d44d765e0 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "e19304fc", - "metadata": {}, + "id": "0b6ca636", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,31 +14,29 @@ }, { "cell_type": "markdown", - "id": "db22c3f3", - "metadata": {}, + "id": "e8840c08", + "metadata": { + "editable": true + }, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", "\n", - "Date: **Nov 1, 2022**\n", - "\n", - "Copyright 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" + "Date: **Week 39**" ] }, { "cell_type": "markdown", - "id": "70506212", - "metadata": {}, + "id": "a9f4d2cf", + "metadata": { + "editable": true + }, "source": [ "## Plan for week 39\n", "\n", - "* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent\n", + "* Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent\n", "\n", - " * [Video of lecture](https://youtu.be/rDBj50Lv3Go)\n", - "\n", - "* Friday: Stochastic Gradient descent with examples and automatic differentiation\n", - "\n", - " * [Video of lecture](https://youtu.be/OH6I_oscwPc)\n", + "* Stochastic Gradient descent with examples and automatic differentiation\n", "\n", "* Reading recommendations:\n", "\n", @@ -51,8 +51,10 @@ }, { "cell_type": "markdown", - "id": "1317c406", - "metadata": {}, + "id": "83479647", + "metadata": { + "editable": true + }, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -70,8 +72,10 @@ }, { "cell_type": "markdown", - "id": "de988ebc", - "metadata": {}, + "id": "1e80db83", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -85,8 +89,10 @@ }, { "cell_type": "markdown", - "id": "c6408cd9", - "metadata": {}, + "id": "44d7e4fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -98,16 +104,20 @@ }, { "cell_type": "markdown", - "id": "5f9ddae7", - "metadata": {}, + "id": "34d2a6f4", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] }, { "cell_type": "markdown", - "id": "140b8948", - "metadata": {}, + "id": "a6f15df5", + "metadata": { + "editable": true + }, "source": [ "## The equations to solve\n", "\n", @@ -120,8 +130,10 @@ }, { "cell_type": "markdown", - "id": "12ecdef5", - "metadata": {}, + "id": "e7571bf9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -130,8 +142,10 @@ }, { "cell_type": "markdown", - "id": "a84ca2bb", - "metadata": {}, + "id": "92063c06", + "metadata": { + "editable": true + }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -139,8 +153,10 @@ }, { "cell_type": "markdown", - "id": "deef1d00", - "metadata": {}, + "id": "976c9856", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -149,16 +165,20 @@ }, { "cell_type": "markdown", - "id": "b46c8220", - "metadata": {}, + "id": "03839482", + "metadata": { + "editable": true + }, "source": [ "This defines what is called the Hessian matrix." ] }, { "cell_type": "markdown", - "id": "94bc14f5", - "metadata": {}, + "id": "f9905a63", + "metadata": { + "editable": true + }, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -169,8 +189,10 @@ }, { "cell_type": "markdown", - "id": "2648c99e", - "metadata": {}, + "id": "88d19971", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -179,16 +201,20 @@ }, { "cell_type": "markdown", - "id": "df77cf5c", - "metadata": {}, + "id": "301f3887", + "metadata": { + "editable": true + }, "source": [ "or in matrix form as" ] }, { "cell_type": "markdown", - "id": "8a7e6efc", - "metadata": {}, + "id": "83164036", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -197,8 +223,10 @@ }, { "cell_type": "markdown", - "id": "350e1f3a", - "metadata": {}, + "id": "d9c42fcc", + "metadata": { + "editable": true + }, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -207,8 +235,10 @@ }, { "cell_type": "markdown", - "id": "e02a25da", - "metadata": {}, + "id": "06d6cf6d", + "metadata": { + "editable": true + }, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -225,8 +255,10 @@ }, { "cell_type": "markdown", - "id": "00253544", - "metadata": {}, + "id": "c41df29b", + "metadata": { + "editable": true + }, "source": [ "## The equations\n", "\n", @@ -239,8 +271,10 @@ }, { "cell_type": "markdown", - "id": "994813f7", - "metadata": {}, + "id": "1e708b4d", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -253,8 +287,10 @@ }, { "cell_type": "markdown", - "id": "82574de2", - "metadata": {}, + "id": "118c1956", + "metadata": { + "editable": true + }, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -262,8 +298,10 @@ }, { "cell_type": "markdown", - "id": "053a7486", - "metadata": {}, + "id": "c437a7bf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -272,16 +310,20 @@ }, { "cell_type": "markdown", - "id": "527459a9", - "metadata": {}, + "id": "4269c5d8", + "metadata": { + "editable": true + }, "source": [ "yielding" ] }, { "cell_type": "markdown", - "id": "9afae8cd", - "metadata": {}, + "id": "000f8372", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -290,16 +332,20 @@ }, { "cell_type": "markdown", - "id": "b2dc015a", - "metadata": {}, + "id": "b4a8ca90", + "metadata": { + "editable": true + }, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] }, { "cell_type": "markdown", - "id": "4a709df9", - "metadata": {}, + "id": "d1629115", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -308,8 +354,10 @@ }, { "cell_type": "markdown", - "id": "31cfe193", - "metadata": {}, + "id": "d9703d98", + "metadata": { + "editable": true + }, "source": [ "## Simple geometric interpretation\n", "\n", @@ -328,8 +376,10 @@ }, { "cell_type": "markdown", - "id": "cc3af9d1", - "metadata": {}, + "id": "956b1426", + "metadata": { + "editable": true + }, "source": [ "## Extending to more than one variable\n", "\n", @@ -339,8 +389,10 @@ }, { "cell_type": "markdown", - "id": "b0d9eaf1", - "metadata": {}, + "id": "628ea4b6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -350,16 +402,20 @@ }, { "cell_type": "markdown", - "id": "1c74bd3b", - "metadata": {}, + "id": "050fe89b", + "metadata": { + "editable": true + }, "source": [ "which we Taylor expand to obtain" ] }, { "cell_type": "markdown", - "id": "7ec82113", - "metadata": {}, + "id": "94ce5a1f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -374,16 +430,20 @@ }, { "cell_type": "markdown", - "id": "ed1cfaa4", - "metadata": {}, + "id": "e0f9817b", + "metadata": { + "editable": true + }, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { "cell_type": "markdown", - "id": "2ac21047", - "metadata": {}, + "id": "ce3d9f6a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -395,16 +455,20 @@ }, { "cell_type": "markdown", - "id": "2063d240", - "metadata": {}, + "id": "bd1b2b82", + "metadata": { + "editable": true + }, "source": [ "we can rephrase Newton's method as" ] }, { "cell_type": "markdown", - "id": "cbf9121f", - "metadata": {}, + "id": "432f95ed", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -415,16 +479,20 @@ }, { "cell_type": "markdown", - "id": "978324df", - "metadata": {}, + "id": "b22896f3", + "metadata": { + "editable": true + }, "source": [ "where we have defined" ] }, { "cell_type": "markdown", - "id": "4b23342f", - "metadata": {}, + "id": "5c1900ef", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -435,8 +503,10 @@ }, { "cell_type": "markdown", - "id": "53ae12e1", - "metadata": {}, + "id": "37826b5b", + "metadata": { + "editable": true + }, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -448,8 +518,10 @@ }, { "cell_type": "markdown", - "id": "2c53554c", - "metadata": {}, + "id": "f05b71c9", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent\n", "\n", @@ -463,8 +535,10 @@ }, { "cell_type": "markdown", - "id": "934c3e8e", - "metadata": {}, + "id": "79497e3e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -473,8 +547,10 @@ }, { "cell_type": "markdown", - "id": "7fb73a2f", - "metadata": {}, + "id": "b5a5f588", + "metadata": { + "editable": true + }, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -485,8 +561,10 @@ }, { "cell_type": "markdown", - "id": "7b685426", - "metadata": {}, + "id": "cfe26d28", + "metadata": { + "editable": true + }, "source": [ "## More on Steepest descent\n", "\n", @@ -498,8 +576,10 @@ }, { "cell_type": "markdown", - "id": "cbbcc943", - "metadata": {}, + "id": "c2458cdf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -508,8 +588,10 @@ }, { "cell_type": "markdown", - "id": "8742fc06", - "metadata": {}, + "id": "62f640a6", + "metadata": { + "editable": true + }, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -517,8 +599,10 @@ }, { "cell_type": "markdown", - "id": "c94adf8d", - "metadata": {}, + "id": "a4074602", + "metadata": { + "editable": true + }, "source": [ "## The ideal\n", "\n", @@ -543,8 +627,10 @@ }, { "cell_type": "markdown", - "id": "3d02afa9", - "metadata": {}, + "id": "ffd18e66", + "metadata": { + "editable": true + }, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -563,8 +649,10 @@ }, { "cell_type": "markdown", - "id": "20c606aa", - "metadata": {}, + "id": "2b9569e0", + "metadata": { + "editable": true + }, "source": [ "## Convex functions\n", "\n", @@ -583,8 +671,10 @@ }, { "cell_type": "markdown", - "id": "f858cfd5", - "metadata": {}, + "id": "54fe0e8b", + "metadata": { + "editable": true + }, "source": [ "## Convex function\n", "\n", @@ -593,8 +683,10 @@ }, { "cell_type": "markdown", - "id": "3d696476", - "metadata": {}, + "id": "196dbe68", + "metadata": { + "editable": true + }, "source": [ "## Conditions on convex functions\n", "\n", @@ -628,8 +720,10 @@ }, { "cell_type": "markdown", - "id": "306dee94", - "metadata": {}, + "id": "7f983eb2", + "metadata": { + "editable": true + }, "source": [ "## More on convex functions\n", "\n", @@ -654,8 +748,10 @@ }, { "cell_type": "markdown", - "id": "3906d87c", - "metadata": {}, + "id": "054cb408", + "metadata": { + "editable": true + }, "source": [ "## Some simple problems\n", "\n", @@ -682,8 +778,10 @@ }, { "cell_type": "markdown", - "id": "5b9f50b0", - "metadata": {}, + "id": "d96c42f0", + "metadata": { + "editable": true + }, "source": [ "## Standard steepest descent\n", "\n", @@ -700,8 +798,10 @@ }, { "cell_type": "markdown", - "id": "58c45902", - "metadata": {}, + "id": "74f954b6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -710,16 +810,20 @@ }, { "cell_type": "markdown", - "id": "e6f32e9c", - "metadata": {}, + "id": "bab38e22", + "metadata": { + "editable": true + }, "source": [ "In the iterative process we end up with a problem like" ] }, { "cell_type": "markdown", - "id": "f6986767", - "metadata": {}, + "id": "d1b87d81", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -728,8 +832,10 @@ }, { "cell_type": "markdown", - "id": "5ca3ae5e", - "metadata": {}, + "id": "2564119f", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -738,8 +844,10 @@ }, { "cell_type": "markdown", - "id": "0df2b40b", - "metadata": {}, + "id": "8aa8dc93", + "metadata": { + "editable": true + }, "source": [ "## Gradient method\n", "\n", @@ -748,8 +856,10 @@ }, { "cell_type": "markdown", - "id": "a8951b6a", - "metadata": {}, + "id": "ce621d40", + "metadata": { + "editable": true + }, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -758,8 +868,10 @@ }, { "cell_type": "markdown", - "id": "e57b2732", - "metadata": {}, + "id": "967e409a", + "metadata": { + "editable": true + }, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -767,8 +879,10 @@ }, { "cell_type": "markdown", - "id": "5ec1a80f", - "metadata": {}, + "id": "5d2ef69c", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "\n", @@ -778,8 +892,10 @@ }, { "cell_type": "markdown", - "id": "072508d4", - "metadata": {}, + "id": "5c5e3bda", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -788,16 +904,20 @@ }, { "cell_type": "markdown", - "id": "dac847f4", - "metadata": {}, + "id": "6cecea3d", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "c3c7a46d", - "metadata": {}, + "id": "096853a4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -806,16 +926,20 @@ }, { "cell_type": "markdown", - "id": "7cff2801", - "metadata": {}, + "id": "edcf059a", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "077c7d80", - "metadata": {}, + "id": "73b9ef7c", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -823,8 +947,10 @@ }, { "cell_type": "markdown", - "id": "1aa22051", - "metadata": {}, + "id": "116c2823", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -833,8 +959,10 @@ }, { "cell_type": "markdown", - "id": "156f3fba", - "metadata": {}, + "id": "00b979dd", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -843,8 +971,10 @@ }, { "cell_type": "markdown", - "id": "cdb97d77", - "metadata": {}, + "id": "8bc10ec6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -853,8 +983,10 @@ }, { "cell_type": "markdown", - "id": "15a7044a", - "metadata": {}, + "id": "4080fe97", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -862,8 +994,10 @@ }, { "cell_type": "markdown", - "id": "cbf9fdb2", - "metadata": {}, + "id": "1f39d1bd", + "metadata": { + "editable": true + }, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -871,8 +1005,10 @@ }, { "cell_type": "markdown", - "id": "891f4d9a", - "metadata": {}, + "id": "7cbf183e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -881,16 +1017,20 @@ }, { "cell_type": "markdown", - "id": "e84b1a57", - "metadata": {}, + "id": "3572dfc2", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "8bc4b9f4", - "metadata": {}, + "id": "530255c0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -899,16 +1039,20 @@ }, { "cell_type": "markdown", - "id": "facdc699", - "metadata": {}, + "id": "92bb5f9f", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "88071807", - "metadata": {}, + "id": "ee402373", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -917,16 +1061,20 @@ }, { "cell_type": "markdown", - "id": "7c71ee58", - "metadata": {}, + "id": "b7ecc434", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "f80c1bd9", - "metadata": {}, + "id": "bcdd6a46", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -935,16 +1083,20 @@ }, { "cell_type": "markdown", - "id": "a69a6141", - "metadata": {}, + "id": "26d2eb03", + "metadata": { + "editable": true + }, "source": [ "leading to the iterative scheme" ] }, { "cell_type": "markdown", - "id": "902457d5", - "metadata": {}, + "id": "b081689d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_{k},\n", @@ -953,8 +1105,10 @@ }, { "cell_type": "markdown", - "id": "e12c7a76", - "metadata": {}, + "id": "5e4dd65b", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent example" ] @@ -962,8 +1116,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "a085458d", - "metadata": {}, + "id": "b5e819e0", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -992,8 +1149,10 @@ }, { "cell_type": "markdown", - "id": "9cb5bdf9", - "metadata": {}, + "id": "bc51c841", + "metadata": { + "editable": true + }, "source": [ "And then as countor plot" ] @@ -1001,8 +1160,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "251fd2a7", - "metadata": {}, + "id": "079d6a5b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1012,8 +1174,10 @@ }, { "cell_type": "markdown", - "id": "64b25b67", - "metadata": {}, + "id": "280c077f", + "metadata": { + "editable": true + }, "source": [ "Find guesses" ] @@ -1021,8 +1185,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "aeb6f371", - "metadata": {}, + "id": "9c0e8d49", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1031,8 +1198,10 @@ }, { "cell_type": "markdown", - "id": "97064a9e", - "metadata": {}, + "id": "6df39dc7", + "metadata": { + "editable": true + }, "source": [ "Run it!" ] @@ -1040,8 +1209,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "02d8c327", - "metadata": {}, + "id": "036ff5ee", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1055,8 +1227,10 @@ }, { "cell_type": "markdown", - "id": "4f22e010", - "metadata": {}, + "id": "2670d5e5", + "metadata": { + "editable": true + }, "source": [ "What happened?" ] @@ -1064,8 +1238,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "980486c5", - "metadata": {}, + "id": "0e68a3cf", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1076,16 +1253,20 @@ }, { "cell_type": "markdown", - "id": "b814c572", - "metadata": {}, + "id": "13118605", + "metadata": { + "editable": true + }, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] }, { "cell_type": "markdown", - "id": "028c7b3e", - "metadata": {}, + "id": "a18cc623", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1096,8 +1277,10 @@ }, { "cell_type": "markdown", - "id": "cd289649", - "metadata": {}, + "id": "b97138f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1106,8 +1289,10 @@ }, { "cell_type": "markdown", - "id": "adcb5d35", - "metadata": {}, + "id": "8a7f0370", + "metadata": { + "editable": true + }, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1115,8 +1300,10 @@ }, { "cell_type": "markdown", - "id": "6181d466", - "metadata": {}, + "id": "f588314c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1125,8 +1312,10 @@ }, { "cell_type": "markdown", - "id": "ff237231", - "metadata": {}, + "id": "7956e6f8", + "metadata": { + "editable": true + }, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1134,8 +1323,10 @@ }, { "cell_type": "markdown", - "id": "e5d58187", - "metadata": {}, + "id": "77230878", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1143,8 +1334,10 @@ }, { "cell_type": "markdown", - "id": "e7ab5477", - "metadata": {}, + "id": "ff8b02db", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1153,16 +1346,20 @@ }, { "cell_type": "markdown", - "id": "dffa1e1a", - "metadata": {}, + "id": "93043c94", + "metadata": { + "editable": true + }, "source": [ "which is zero unless $i=j$." ] }, { "cell_type": "markdown", - "id": "35b985cd", - "metadata": {}, + "id": "4879c307", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1171,8 +1368,10 @@ }, { "cell_type": "markdown", - "id": "a1cd557c", - "metadata": {}, + "id": "dd55a064", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1181,8 +1380,10 @@ }, { "cell_type": "markdown", - "id": "13378a74", - "metadata": {}, + "id": "6d698cb0", + "metadata": { + "editable": true + }, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1191,8 +1392,10 @@ }, { "cell_type": "markdown", - "id": "77acd0cf", - "metadata": {}, + "id": "ff388fc5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1201,8 +1404,10 @@ }, { "cell_type": "markdown", - "id": "89382545", - "metadata": {}, + "id": "1e9b1ec2", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1210,8 +1415,10 @@ }, { "cell_type": "markdown", - "id": "efd2d8ae", - "metadata": {}, + "id": "7c9d1ee3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1220,16 +1427,20 @@ }, { "cell_type": "markdown", - "id": "48f4c6b5", - "metadata": {}, + "id": "8d547db3", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] }, { "cell_type": "markdown", - "id": "cacf2c51", - "metadata": {}, + "id": "2f4a5adc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1238,16 +1449,20 @@ }, { "cell_type": "markdown", - "id": "9b96eea3", - "metadata": {}, + "id": "b7fd0f39", + "metadata": { + "editable": true + }, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] }, { "cell_type": "markdown", - "id": "cea80d64", - "metadata": {}, + "id": "e70cf1f1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1256,8 +1471,10 @@ }, { "cell_type": "markdown", - "id": "3ce4ef01", - "metadata": {}, + "id": "f9e42581", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1274,8 +1491,10 @@ }, { "cell_type": "markdown", - "id": "02fcf3fc", - "metadata": {}, + "id": "cb104480", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1284,16 +1503,20 @@ }, { "cell_type": "markdown", - "id": "ea42875a", - "metadata": {}, + "id": "1aa99016", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "87f3507a", - "metadata": {}, + "id": "c6005530", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1302,16 +1525,20 @@ }, { "cell_type": "markdown", - "id": "d7d840fc", - "metadata": {}, + "id": "f09ba534", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "0b29e7e2", - "metadata": {}, + "id": "8ee4ad71", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1319,8 +1546,10 @@ }, { "cell_type": "markdown", - "id": "167176f2", - "metadata": {}, + "id": "f1235226", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1329,8 +1558,10 @@ }, { "cell_type": "markdown", - "id": "c60f64c5", - "metadata": {}, + "id": "13ad7eb8", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1339,8 +1570,10 @@ }, { "cell_type": "markdown", - "id": "1b207909", - "metadata": {}, + "id": "c66691cb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1349,8 +1582,10 @@ }, { "cell_type": "markdown", - "id": "cfee159c", - "metadata": {}, + "id": "b1dafe82", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1360,8 +1595,10 @@ }, { "cell_type": "markdown", - "id": "7002d653", - "metadata": {}, + "id": "4603d128", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1369,8 +1606,10 @@ }, { "cell_type": "markdown", - "id": "e13d8483", - "metadata": {}, + "id": "bfa292fd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1379,8 +1618,10 @@ }, { "cell_type": "markdown", - "id": "be58b0e1", - "metadata": {}, + "id": "636e79c4", + "metadata": { + "editable": true + }, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1393,8 +1634,10 @@ }, { "cell_type": "markdown", - "id": "46c71245", - "metadata": {}, + "id": "6d3e71b9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1403,8 +1646,10 @@ }, { "cell_type": "markdown", - "id": "daa7b7da", - "metadata": {}, + "id": "9f6b7605", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1412,8 +1657,10 @@ }, { "cell_type": "markdown", - "id": "ab13c237", - "metadata": {}, + "id": "90c617b8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1422,16 +1669,20 @@ }, { "cell_type": "markdown", - "id": "b4416925", - "metadata": {}, + "id": "690e77f7", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "572d4184", - "metadata": {}, + "id": "ebe87bbf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1440,16 +1691,20 @@ }, { "cell_type": "markdown", - "id": "3f1c3d2b", - "metadata": {}, + "id": "284ebebe", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "132b319d", - "metadata": {}, + "id": "4a67d86f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1458,16 +1713,20 @@ }, { "cell_type": "markdown", - "id": "24521e78", - "metadata": {}, + "id": "bfa0b7a4", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "1f98d52a", - "metadata": {}, + "id": "11b5ff1d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1476,8 +1735,10 @@ }, { "cell_type": "markdown", - "id": "6764a095", - "metadata": {}, + "id": "1a68b320", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our first homework\n", "\n", @@ -1498,8 +1759,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "097ba45d", - "metadata": {}, + "id": "f6b45066", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1508,8 +1772,10 @@ }, { "cell_type": "markdown", - "id": "7e002e40", - "metadata": {}, + "id": "941ec30c", + "metadata": { + "editable": true + }, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1517,8 +1783,10 @@ }, { "cell_type": "markdown", - "id": "8c6eb2d8", - "metadata": {}, + "id": "b2c795b2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1527,16 +1795,20 @@ }, { "cell_type": "markdown", - "id": "306c4dc4", - "metadata": {}, + "id": "aaa16809", + "metadata": { + "editable": true + }, "source": [ "such that" ] }, { "cell_type": "markdown", - "id": "15a602aa", - "metadata": {}, + "id": "b9010811", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1545,8 +1817,10 @@ }, { "cell_type": "markdown", - "id": "a0ad20d1", - "metadata": {}, + "id": "7a496ec5", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent example\n", "\n", @@ -1557,8 +1831,10 @@ }, { "cell_type": "markdown", - "id": "443f2c5f", - "metadata": {}, + "id": "b96688aa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1571,16 +1847,20 @@ }, { "cell_type": "markdown", - "id": "fa5b73b0", - "metadata": {}, + "id": "b10f96e6", + "metadata": { + "editable": true + }, "source": [ "The cost/loss/risk function is given by (" ] }, { "cell_type": "markdown", - "id": "2be1c346", - "metadata": {}, + "id": "8ac0e224", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1589,16 +1869,20 @@ }, { "cell_type": "markdown", - "id": "0716f795", - "metadata": {}, + "id": "819d5d1c", + "metadata": { + "editable": true + }, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] }, { "cell_type": "markdown", - "id": "d283d92e", - "metadata": {}, + "id": "ab659bad", + "metadata": { + "editable": true + }, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -1607,8 +1891,10 @@ }, { "cell_type": "markdown", - "id": "87219710", - "metadata": {}, + "id": "8c35e5a2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1619,16 +1905,20 @@ }, { "cell_type": "markdown", - "id": "40cb318c", - "metadata": {}, + "id": "0e3d77e1", + "metadata": { + "editable": true + }, "source": [ "where $X$ is the design matrix defined above." ] }, { "cell_type": "markdown", - "id": "51e172a8", - "metadata": {}, + "id": "b0b9b496", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -1636,8 +1926,10 @@ }, { "cell_type": "markdown", - "id": "7a8d39e9", - "metadata": {}, + "id": "c102ee11", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1649,16 +1941,20 @@ }, { "cell_type": "markdown", - "id": "5794a187", - "metadata": {}, + "id": "32dfcdd7", + "metadata": { + "editable": true + }, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] }, { "cell_type": "markdown", - "id": "0b3a3a65", - "metadata": {}, + "id": "1f905bde", + "metadata": { + "editable": true + }, "source": [ "## Simple program\n", "\n", @@ -1667,8 +1963,10 @@ }, { "cell_type": "markdown", - "id": "24aa13fa", - "metadata": {}, + "id": "300e8838", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1677,8 +1975,10 @@ }, { "cell_type": "markdown", - "id": "9ecfc1e6", - "metadata": {}, + "id": "d28d496c", + "metadata": { + "editable": true + }, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -1690,8 +1990,10 @@ }, { "cell_type": "markdown", - "id": "02725671", - "metadata": {}, + "id": "445d56e6", + "metadata": { + "editable": true + }, "source": [ "## Gradient Descent Example\n", "\n", @@ -1701,8 +2003,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "cd0162fa", - "metadata": {}, + "id": "5b013b95", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1755,8 +2060,10 @@ }, { "cell_type": "markdown", - "id": "1fad0195", - "metadata": {}, + "id": "85d33a34", + "metadata": { + "editable": true + }, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -1764,8 +2071,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "92a78685", - "metadata": {}, + "id": "ef815a7a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -1788,8 +2098,10 @@ }, { "cell_type": "markdown", - "id": "1401c67e", - "metadata": {}, + "id": "b4bd8c37", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -1798,8 +2110,10 @@ }, { "cell_type": "markdown", - "id": "ba68372f", - "metadata": {}, + "id": "58ac27e8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -1808,16 +2122,20 @@ }, { "cell_type": "markdown", - "id": "10caca16", - "metadata": {}, + "id": "c146f500", + "metadata": { + "editable": true + }, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] }, { "cell_type": "markdown", - "id": "92d72d1b", - "metadata": {}, + "id": "afdf568c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1828,16 +2146,20 @@ }, { "cell_type": "markdown", - "id": "6c020c85", - "metadata": {}, + "id": "0e03b0f2", + "metadata": { + "editable": true + }, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] }, { "cell_type": "markdown", - "id": "36b002d0", - "metadata": {}, + "id": "f7d35405", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -1846,8 +2168,10 @@ }, { "cell_type": "markdown", - "id": "2d89c0ad", - "metadata": {}, + "id": "eee985e8", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -1855,8 +2179,10 @@ }, { "cell_type": "markdown", - "id": "d85e005a", - "metadata": {}, + "id": "ebbe199f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1868,8 +2194,10 @@ }, { "cell_type": "markdown", - "id": "3e83b611", - "metadata": {}, + "id": "efa70eb1", + "metadata": { + "editable": true + }, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -1880,8 +2208,10 @@ }, { "cell_type": "markdown", - "id": "3eaea477", - "metadata": {}, + "id": "2dc242be", + "metadata": { + "editable": true + }, "source": [ "## Program example for gradient descent with Ridge Regression" ] @@ -1889,8 +2219,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "1b451328", - "metadata": {}, + "id": "d31b5b9a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from random import random, seed\n", @@ -1947,8 +2280,10 @@ }, { "cell_type": "markdown", - "id": "f7a052a3", - "metadata": {}, + "id": "29a18c30", + "metadata": { + "editable": true + }, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -1967,8 +2302,10 @@ }, { "cell_type": "markdown", - "id": "a1ae154d", - "metadata": {}, + "id": "5ba5a047", + "metadata": { + "editable": true + }, "source": [ "## Improving gradient descent with momentum\n", "\n", @@ -1978,8 +2315,11 @@ { "cell_type": "code", "execution_count": 10, - "id": "4b5802c8", - "metadata": {}, + "id": "946e5b06", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from numpy import asarray\n", @@ -2041,8 +2381,10 @@ }, { "cell_type": "markdown", - "id": "937ec20c", - "metadata": {}, + "id": "0100a73f", + "metadata": { + "editable": true + }, "source": [ "## Same code but now with momentum gradient descent" ] @@ -2050,8 +2392,11 @@ { "cell_type": "code", "execution_count": 11, - "id": "1c70ef39", - "metadata": {}, + "id": "6cbbf9f1", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from numpy import asarray\n", @@ -2121,8 +2466,10 @@ }, { "cell_type": "markdown", - "id": "e3f3f35b", - "metadata": {}, + "id": "0e599f71", + "metadata": { + "editable": true + }, "source": [ "## Overview video on Stochastic Gradient Descent\n", "\n", @@ -2131,8 +2478,10 @@ }, { "cell_type": "markdown", - "id": "66e49d9f", - "metadata": {}, + "id": "5cde0b8b", + "metadata": { + "editable": true + }, "source": [ "## Batches and mini-batches\n", "\n", @@ -2150,8 +2499,10 @@ }, { "cell_type": "markdown", - "id": "b001bba9", - "metadata": {}, + "id": "7b8e3b3f", + "metadata": { + "editable": true + }, "source": [ "## Stochastic Gradient Descent (SGD)\n", "\n", @@ -2180,8 +2531,10 @@ }, { "cell_type": "markdown", - "id": "f6fcb791", - "metadata": {}, + "id": "c1786f4c", + "metadata": { + "editable": true + }, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2195,8 +2548,10 @@ }, { "cell_type": "markdown", - "id": "db94a1da", - "metadata": {}, + "id": "4462b6fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2206,8 +2561,10 @@ }, { "cell_type": "markdown", - "id": "c4829ebe", - "metadata": {}, + "id": "09b2f507", + "metadata": { + "editable": true + }, "source": [ "## Computation of gradients\n", "\n", @@ -2217,8 +2574,10 @@ }, { "cell_type": "markdown", - "id": "007d0669", - "metadata": {}, + "id": "31510e29", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2228,8 +2587,10 @@ }, { "cell_type": "markdown", - "id": "45e22e19", - "metadata": {}, + "id": "78f3f26d", + "metadata": { + "editable": true + }, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2240,8 +2601,10 @@ }, { "cell_type": "markdown", - "id": "603a3d46", - "metadata": {}, + "id": "bb53530d", + "metadata": { + "editable": true + }, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2260,8 +2623,10 @@ }, { "cell_type": "markdown", - "id": "a65a522a", - "metadata": {}, + "id": "79d494c4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2273,8 +2638,10 @@ }, { "cell_type": "markdown", - "id": "190a4a82", - "metadata": {}, + "id": "1a6f31f3", + "metadata": { + "editable": true + }, "source": [ "## The gradient step\n", "\n", @@ -2283,8 +2650,10 @@ }, { "cell_type": "markdown", - "id": "2902d185", - "metadata": {}, + "id": "770c4a2b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2294,8 +2663,10 @@ }, { "cell_type": "markdown", - "id": "a8a514b5", - "metadata": {}, + "id": "640f9b47", + "metadata": { + "editable": true + }, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2306,8 +2677,10 @@ }, { "cell_type": "markdown", - "id": "8ee6e9f0", - "metadata": {}, + "id": "772329af", + "metadata": { + "editable": true + }, "source": [ "## Simple example code" ] @@ -2315,8 +2688,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "3a795723", - "metadata": {}, + "id": "cdb099f5", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2337,8 +2713,10 @@ }, { "cell_type": "markdown", - "id": "0c179bd5", - "metadata": {}, + "id": "40e10ef6", + "metadata": { + "editable": true + }, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2351,8 +2729,10 @@ }, { "cell_type": "markdown", - "id": "f111b71f", - "metadata": {}, + "id": "1d19e5a8", + "metadata": { + "editable": true + }, "source": [ "## When do we stop?\n", "\n", @@ -2370,8 +2750,10 @@ }, { "cell_type": "markdown", - "id": "b81892de", - "metadata": {}, + "id": "7fab9fc2", + "metadata": { + "editable": true + }, "source": [ "## Slightly different approach\n", "\n", @@ -2388,8 +2770,10 @@ }, { "cell_type": "markdown", - "id": "f4aaa434", - "metadata": {}, + "id": "a4c22706", + "metadata": { + "editable": true + }, "source": [ "## Time decay rate\n", "\n", @@ -2405,8 +2789,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "c2713c7c", - "metadata": {}, + "id": "d7a2b4d8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2437,8 +2824,10 @@ }, { "cell_type": "markdown", - "id": "b9ffc236", - "metadata": {}, + "id": "09bb98a4", + "metadata": { + "editable": true + }, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2448,8 +2837,11 @@ { "cell_type": "code", "execution_count": 14, - "id": "3dd8c731", - "metadata": {}, + "id": "a1900f25", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -2522,8 +2914,10 @@ }, { "cell_type": "markdown", - "id": "522a784b", - "metadata": {}, + "id": "c94f1d84", + "metadata": { + "editable": true + }, "source": [ "## Replace or not\n", "\n", @@ -2535,8 +2929,10 @@ }, { "cell_type": "markdown", - "id": "fd260edf", - "metadata": {}, + "id": "2483e9ce", + "metadata": { + "editable": true + }, "source": [ "## Momentum based GD\n", "\n", @@ -2548,8 +2944,10 @@ }, { "cell_type": "markdown", - "id": "0111d119", - "metadata": {}, + "id": "983d9d50", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", @@ -2558,8 +2956,10 @@ }, { "cell_type": "markdown", - "id": "7786ba69", - "metadata": {}, + "id": "f3f609a3", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2574,8 +2974,10 @@ }, { "cell_type": "markdown", - "id": "a1aa5c70", - "metadata": {}, + "id": "f411d2ea", + "metadata": { + "editable": true + }, "source": [ "where we have introduced a momentum parameter $\\gamma$, with\n", "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", @@ -2591,8 +2993,10 @@ }, { "cell_type": "markdown", - "id": "7d303a7b", - "metadata": {}, + "id": "4ab2c5a8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", @@ -2601,16 +3005,20 @@ }, { "cell_type": "markdown", - "id": "af7db9f3", - "metadata": {}, + "id": "3f7480b7", + "metadata": { + "editable": true + }, "source": [ "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." ] }, { "cell_type": "markdown", - "id": "d853b42d", - "metadata": {}, + "id": "e600ba28", + "metadata": { + "editable": true + }, "source": [ "## More on momentum based approaches\n", "\n", @@ -2623,8 +3031,10 @@ }, { "cell_type": "markdown", - "id": "878642cf", - "metadata": {}, + "id": "a8e40454", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", @@ -2633,16 +3043,20 @@ }, { "cell_type": "markdown", - "id": "a41c7a65", - "metadata": {}, + "id": "1a241786", + "metadata": { + "editable": true + }, "source": [ "We can discretize this equation in the usual way to get" ] }, { "cell_type": "markdown", - "id": "dcc2658e", - "metadata": {}, + "id": "ae87e69e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", @@ -2651,16 +3065,20 @@ }, { "cell_type": "markdown", - "id": "02660dc6", - "metadata": {}, + "id": "4ceecc15", + "metadata": { + "editable": true + }, "source": [ "Rearranging this equation, we can rewrite this as" ] }, { "cell_type": "markdown", - "id": "9c309fa8", - "metadata": {}, + "id": "882ffbf4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", @@ -2669,8 +3087,10 @@ }, { "cell_type": "markdown", - "id": "ce489766", - "metadata": {}, + "id": "2ae9ed9e", + "metadata": { + "editable": true + }, "source": [ "## Momentum parameter\n", "\n", @@ -2683,8 +3103,10 @@ }, { "cell_type": "markdown", - "id": "ae256ffc", - "metadata": {}, + "id": "d5acf4bb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", @@ -2693,8 +3115,10 @@ }, { "cell_type": "markdown", - "id": "fe64c5a3", - "metadata": {}, + "id": "d20d63ac", + "metadata": { + "editable": true + }, "source": [ "Thus, as the name suggests, the momentum parameter is proportional to\n", "the mass of the particle and effectively provides inertia.\n", @@ -2724,8 +3148,10 @@ }, { "cell_type": "markdown", - "id": "7d736932", - "metadata": {}, + "id": "a8405106", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", @@ -2734,8 +3160,10 @@ }, { "cell_type": "markdown", - "id": "980eb061", - "metadata": {}, + "id": "58509b9c", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2750,16 +3178,20 @@ }, { "cell_type": "markdown", - "id": "6aed8662", - "metadata": {}, + "id": "1f487a62", + "metadata": { + "editable": true + }, "source": [ "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$." ] }, { "cell_type": "markdown", - "id": "fca1f3cf", - "metadata": {}, + "id": "85af90cc", + "metadata": { + "editable": true + }, "source": [ "## Second moment of the gradient\n", "\n", @@ -2787,8 +3219,10 @@ }, { "cell_type": "markdown", - "id": "63a05ce5", - "metadata": {}, + "id": "6763d293", + "metadata": { + "editable": true + }, "source": [ "## RMS prop\n", "\n", @@ -2800,8 +3234,10 @@ }, { "cell_type": "markdown", - "id": "0ce12314", - "metadata": {}, + "id": "6269a700", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2816,8 +3252,10 @@ }, { "cell_type": "markdown", - "id": "ffb6f313", - "metadata": {}, + "id": "3b8071de", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", @@ -2826,8 +3264,10 @@ }, { "cell_type": "markdown", - "id": "0d5d3b11", - "metadata": {}, + "id": "6ea30bc4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", @@ -2836,8 +3276,10 @@ }, { "cell_type": "markdown", - "id": "ac44eacb", - "metadata": {}, + "id": "2022c06c", + "metadata": { + "editable": true + }, "source": [ "where $\\beta$ controls the averaging time of the second moment and is\n", "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", @@ -2852,8 +3294,10 @@ }, { "cell_type": "markdown", - "id": "a9b91c67", - "metadata": {}, + "id": "7a0404a9", + "metadata": { + "editable": true + }, "source": [ "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", "\n", @@ -2879,8 +3323,10 @@ }, { "cell_type": "markdown", - "id": "a0535b70", - "metadata": {}, + "id": "5f8513bf", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2895,8 +3341,10 @@ }, { "cell_type": "markdown", - "id": "37fae235", - "metadata": {}, + "id": "144b23a8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", @@ -2905,8 +3353,10 @@ }, { "cell_type": "markdown", - "id": "24299be0", - "metadata": {}, + "id": "8cd279b7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", @@ -2915,8 +3365,10 @@ }, { "cell_type": "markdown", - "id": "a55ef5dd", - "metadata": {}, + "id": "03206a26", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", @@ -2925,8 +3377,10 @@ }, { "cell_type": "markdown", - "id": "55ebbb4a", - "metadata": {}, + "id": "6e1d0345", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", @@ -2935,8 +3389,10 @@ }, { "cell_type": "markdown", - "id": "6a75cd0f", - "metadata": {}, + "id": "f50703d3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", @@ -2945,8 +3401,10 @@ }, { "cell_type": "markdown", - "id": "bad31ff5", - "metadata": {}, + "id": "4eb84630", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2960,8 +3418,10 @@ }, { "cell_type": "markdown", - "id": "fd7ae8af", - "metadata": {}, + "id": "a64d776b", + "metadata": { + "editable": true + }, "source": [ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", @@ -2977,8 +3437,10 @@ }, { "cell_type": "markdown", - "id": "ebb9ba37", - "metadata": {}, + "id": "cb3f4d29", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", @@ -2987,8 +3449,10 @@ }, { "cell_type": "markdown", - "id": "c1a68fa5", - "metadata": {}, + "id": "4d5878ed", + "metadata": { + "editable": true + }, "source": [ "## Algorithms and codes for Adagrad, RMSprop and Adam\n", "\n", @@ -2999,8 +3463,10 @@ }, { "cell_type": "markdown", - "id": "4eaf4055", - "metadata": {}, + "id": "addbba75", + "metadata": { + "editable": true + }, "source": [ "## Practical tips\n", "\n", @@ -3017,8 +3483,10 @@ }, { "cell_type": "markdown", - "id": "e5ca63c5", - "metadata": {}, + "id": "2707828a", + "metadata": { + "editable": true + }, "source": [ "## Automatic differentiation\n", "\n", @@ -3053,8 +3521,10 @@ }, { "cell_type": "markdown", - "id": "2f8a51f3", - "metadata": {}, + "id": "c879d74d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x) = \\sin\\left(2\\pi x + x^2\\right)\n", @@ -3063,16 +3533,20 @@ }, { "cell_type": "markdown", - "id": "cb951ad3", - "metadata": {}, + "id": "50c78afa", + "metadata": { + "editable": true + }, "source": [ "which has the following derivative" ] }, { "cell_type": "markdown", - "id": "db24016c", - "metadata": {}, + "id": "2e31b945", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n", @@ -3081,8 +3555,10 @@ }, { "cell_type": "markdown", - "id": "1bdf0cb9", - "metadata": {}, + "id": "fbc53258", + "metadata": { + "editable": true + }, "source": [ "Using **autograd** we have" ] @@ -3090,8 +3566,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "6fb5fe82", - "metadata": {}, + "id": "8559136e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3132,8 +3611,10 @@ }, { "cell_type": "markdown", - "id": "0bcf5cb7", - "metadata": {}, + "id": "7c2fb26b", + "metadata": { + "editable": true + }, "source": [ "## Using autograd\n", "\n", @@ -3147,8 +3628,11 @@ { "cell_type": "code", "execution_count": 16, - "id": "ae500933", - "metadata": {}, + "id": "658e24b9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3172,8 +3656,10 @@ }, { "cell_type": "markdown", - "id": "f92c5e9e", - "metadata": {}, + "id": "5449cca9", + "metadata": { + "editable": true + }, "source": [ "## Autograd with more complicated functions\n", "\n", @@ -3185,8 +3671,11 @@ { "cell_type": "code", "execution_count": 17, - "id": "dacd393c", - "metadata": {}, + "id": "2c3b79a8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3226,16 +3715,20 @@ }, { "cell_type": "markdown", - "id": "96661f43", - "metadata": {}, + "id": "5bd26ddb", + "metadata": { + "editable": true + }, "source": [ "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable." ] }, { "cell_type": "markdown", - "id": "bf5240fb", - "metadata": {}, + "id": "1eb6a361", + "metadata": { + "editable": true + }, "source": [ "## More complicated functions using the elements of their arguments directly" ] @@ -3243,8 +3736,11 @@ { "cell_type": "code", "execution_count": 18, - "id": "3b1897e7", - "metadata": {}, + "id": "700b1e86", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3268,8 +3764,10 @@ }, { "cell_type": "markdown", - "id": "3912be8c", - "metadata": {}, + "id": "9f2fa4a1", + "metadata": { + "editable": true + }, "source": [ "Note that in this case, when sending an array as input argument, the\n", "output from Autograd is another array. This is the true gradient of\n", @@ -3281,8 +3779,10 @@ }, { "cell_type": "markdown", - "id": "4992b4b5", - "metadata": {}, + "id": "c5b1f850", + "metadata": { + "editable": true + }, "source": [ "## Functions using mathematical functions from Numpy" ] @@ -3290,8 +3790,11 @@ { "cell_type": "code", "execution_count": 19, - "id": "e05afdd4", - "metadata": {}, + "id": "c6302980", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3315,8 +3818,10 @@ }, { "cell_type": "markdown", - "id": "55bb0b58", - "metadata": {}, + "id": "e3e526c6", + "metadata": { + "editable": true + }, "source": [ "## More autograd" ] @@ -3324,8 +3829,11 @@ { "cell_type": "code", "execution_count": 20, - "id": "0839da1a", - "metadata": {}, + "id": "744be574", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3346,8 +3854,10 @@ }, { "cell_type": "markdown", - "id": "4829f20d", - "metadata": {}, + "id": "50c2ebd2", + "metadata": { + "editable": true + }, "source": [ "## And with loops" ] @@ -3355,8 +3865,11 @@ { "cell_type": "code", "execution_count": 21, - "id": "4fa0bc19", - "metadata": {}, + "id": "20475a58", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3388,8 +3901,11 @@ { "cell_type": "code", "execution_count": 22, - "id": "b52bf0c9", - "metadata": {}, + "id": "d6aec0be", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3405,8 +3921,10 @@ }, { "cell_type": "markdown", - "id": "01cf1786", - "metadata": {}, + "id": "16863bc8", + "metadata": { + "editable": true + }, "source": [ "## Using recursion" ] @@ -3414,8 +3932,11 @@ { "cell_type": "code", "execution_count": 23, - "id": "746b92c4", - "metadata": {}, + "id": "417f5fe0", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3449,16 +3970,20 @@ }, { "cell_type": "markdown", - "id": "af2b252d", - "metadata": {}, + "id": "a971daec", + "metadata": { + "editable": true + }, "source": [ "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input." ] }, { "cell_type": "markdown", - "id": "f008f45c", - "metadata": {}, + "id": "99d11189", + "metadata": { + "editable": true + }, "source": [ "## Unsupported functions\n", "Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n", @@ -3469,8 +3994,11 @@ { "cell_type": "code", "execution_count": 24, - "id": "89111774", - "metadata": {}, + "id": "aa1fbaa1", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3488,16 +4016,20 @@ }, { "cell_type": "markdown", - "id": "89134961", - "metadata": {}, + "id": "23ba2063", + "metadata": { + "editable": true + }, "source": [ "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible." ] }, { "cell_type": "markdown", - "id": "9ccf91b8", - "metadata": {}, + "id": "d0691879", + "metadata": { + "editable": true + }, "source": [ "## The syntax a.dot(b) when finding the dot product" ] @@ -3505,8 +4037,11 @@ { "cell_type": "code", "execution_count": 25, - "id": "42f2b0b1", - "metadata": {}, + "id": "d9741db5", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3524,8 +4059,10 @@ }, { "cell_type": "markdown", - "id": "f620e717", - "metadata": {}, + "id": "c765b721", + "metadata": { + "editable": true + }, "source": [ "Here we are told that the 'dot' function does not belong to Autograd's\n", "version of a Numpy array. To overcome this, an alternative syntax\n", @@ -3535,8 +4072,11 @@ { "cell_type": "code", "execution_count": 26, - "id": "42bfe3e0", - "metadata": {}, + "id": "c1837dcc", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3557,8 +4097,10 @@ }, { "cell_type": "markdown", - "id": "f2b6bd2b", - "metadata": {}, + "id": "1decd6b6", + "metadata": { + "editable": true + }, "source": [ "## Recommended to avoid\n", "The documentation recommends to avoid inplace operations such as" @@ -3567,8 +4109,11 @@ { "cell_type": "code", "execution_count": 27, - "id": "66ee5069", - "metadata": {}, + "id": "fb7fadd3", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "a += b\n", @@ -3579,8 +4124,10 @@ }, { "cell_type": "markdown", - "id": "f89ccece", - "metadata": {}, + "id": "d1556c09", + "metadata": { + "editable": true + }, "source": [ "## Using Autograd with OLS\n", "\n", @@ -3592,8 +4139,11 @@ { "cell_type": "code", "execution_count": 28, - "id": "6914ba39", - "metadata": {}, + "id": "f500b737", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients for OLS\n", @@ -3649,8 +4199,10 @@ }, { "cell_type": "markdown", - "id": "351ca93f", - "metadata": {}, + "id": "753b1d28", + "metadata": { + "editable": true + }, "source": [ "## Same code but now with momentum gradient descent" ] @@ -3658,8 +4210,11 @@ { "cell_type": "code", "execution_count": 29, - "id": "e455eb87", - "metadata": {}, + "id": "5a8981f6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients for OLS\n", @@ -3719,8 +4274,10 @@ }, { "cell_type": "markdown", - "id": "ab6b7d5a", - "metadata": {}, + "id": "25128c2c", + "metadata": { + "editable": true + }, "source": [ "## But noen of these can compete with Newton's method" ] @@ -3728,8 +4285,11 @@ { "cell_type": "code", "execution_count": 30, - "id": "dfe25d60", - "metadata": {}, + "id": "8a030f2d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Newton's method\n", @@ -3774,8 +4334,10 @@ }, { "cell_type": "markdown", - "id": "da215bb2", - "metadata": {}, + "id": "d726e84e", + "metadata": { + "editable": true + }, "source": [ "## Including Stochastic Gradient Descent with Autograd\n", "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**." @@ -3784,8 +4346,11 @@ { "cell_type": "code", "execution_count": 31, - "id": "3a15ed51", - "metadata": {}, + "id": "1d1f12f8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using SGD\n", @@ -3865,8 +4430,10 @@ }, { "cell_type": "markdown", - "id": "ec160328", - "metadata": {}, + "id": "a4615a81", + "metadata": { + "editable": true + }, "source": [ "## Same code but now with momentum gradient descent" ] @@ -3874,8 +4441,11 @@ { "cell_type": "code", "execution_count": 32, - "id": "4465e491", - "metadata": {}, + "id": "4ea0f0d4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using SGD\n", @@ -3949,8 +4519,10 @@ }, { "cell_type": "markdown", - "id": "65a7e3fd", - "metadata": {}, + "id": "2e5b6320", + "metadata": { + "editable": true + }, "source": [ "## Similar (second order function now) problem but now with AdaGrad" ] @@ -3958,8 +4530,11 @@ { "cell_type": "code", "execution_count": 33, - "id": "fbe692b5", - "metadata": {}, + "id": "b6cd6b48", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n", @@ -4014,16 +4589,20 @@ }, { "cell_type": "markdown", - "id": "f2f38754", - "metadata": {}, + "id": "e3bdc8ab", + "metadata": { + "editable": true + }, "source": [ "Running this code we note an almost perfect agreement with the results from matrix inversion." ] }, { "cell_type": "markdown", - "id": "d4eba163", - "metadata": {}, + "id": "6b81d426", + "metadata": { + "editable": true + }, "source": [ "## RMSprop for adaptive learning rate with Stochastic Gradient Descent" ] @@ -4031,8 +4610,11 @@ { "cell_type": "code", "execution_count": 34, - "id": "a580c4a5", - "metadata": {}, + "id": "dd810f17", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n", @@ -4093,8 +4675,10 @@ }, { "cell_type": "markdown", - "id": "0aa9de47", - "metadata": {}, + "id": "1482fbca", + "metadata": { + "editable": true + }, "source": [ "## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)" ] @@ -4102,8 +4686,11 @@ { "cell_type": "code", "execution_count": 35, - "id": "987e868e", - "metadata": {}, + "id": "510c96db", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n", @@ -4169,8 +4756,10 @@ }, { "cell_type": "markdown", - "id": "7a1aabcc", - "metadata": {}, + "id": "64368447", + "metadata": { + "editable": true + }, "source": [ "## And Logistic Regression" ] @@ -4178,8 +4767,11 @@ { "cell_type": "code", "execution_count": 36, - "id": "c44e7a69", - "metadata": {}, + "id": "a54f6964", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4219,8 +4811,10 @@ }, { "cell_type": "markdown", - "id": "0a479a8e", - "metadata": {}, + "id": "df97ad5c", + "metadata": { + "editable": true + }, "source": [ "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n", "\n", @@ -4235,18 +4829,13 @@ }, { "cell_type": "code", - "execution_count": 1, - "id": "e8fdc7be", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[0.25 0.19661197 0.10499357]\n" - ] - } - ], + "execution_count": 37, + "id": "cf6f0fdc", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import jax.numpy as jnp\n", "from jax import grad, jit, vmap\n", @@ -4258,35 +4847,9 @@ "derivative_fn = grad(sum_logistic)\n", "print(derivative_fn(x_small))" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1a930655", - "metadata": {}, - "outputs": [], - "source": [] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.10" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index a61d07fa7..fbb6a344f 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -1,14 +1,13 @@ TITLE: Week 39: Optimization and Gradient Methods AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University -DATE: today +DATE: Week 39 !split ===== Plan for week 39 ===== -* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent - * "Video of lecture":"https://youtu.be/rDBj50Lv3Go" -* Friday: Stochastic Gradient descent with examples and automatic differentiation - * "Video of lecture":"https://youtu.be/OH6I_oscwPc" +* Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent +* Stochastic Gradient descent with examples and automatic differentiation + * Reading recommendations: See "lecture notes for week 39":"https://compphysics.github.io/MachineLearning/doc/web/course.html".