From b83d90df2b0419d07a2f44c8b34d7119810088bd Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Tue, 17 Sep 2024 22:17:26 +0200 Subject: [PATCH] update week 39 --- doc/LectureNotes/exercisesweek39.ipynb | 59 + doc/pub/week39/html/week39-bs.html | 10 +- doc/pub/week39/html/week39-reveal.html | 50 +- doc/pub/week39/html/week39-solarized.html | 44 +- doc/pub/week39/html/week39.html | 44 +- doc/pub/week39/ipynb/ipynb-week39-src.tar.gz | Bin 191 -> 193 bytes doc/pub/week39/ipynb/week39.ipynb | 1789 ++++++++++++------ doc/src/week39/exercisesweek39.do.txt | 8 +- doc/src/week39/week39.do.txt | 22 +- 9 files changed, 1336 insertions(+), 690 deletions(-) create mode 100644 doc/LectureNotes/exercisesweek39.ipynb diff --git a/doc/LectureNotes/exercisesweek39.ipynb b/doc/LectureNotes/exercisesweek39.ipynb new file mode 100644 index 000000000..b80557f7f --- /dev/null +++ b/doc/LectureNotes/exercisesweek39.ipynb @@ -0,0 +1,59 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "263a45c0", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "155995fb", + "metadata": { + "editable": true + }, + "source": [ + "# Exercises week 39\n", + "**September 23-27, 2024**\n", + "\n", + "Date: **Deadline is Friday September 27 at midnight**" + ] + }, + { + "cell_type": "markdown", + "id": "402606f9", + "metadata": { + "editable": true + }, + "source": [ + "## Overarching aims of the exercises this week\n", + "\n", + "The aim of the exercises this week is to aid you in getting started\n", + "with writing the report. This will be discussed during the lab\n", + "sessions as well. \n", + "\n", + "A general guideline can be found at .\n", + "\n", + "Similarly, an example of an earlier project can be found at \n", + "\n", + "Your task this week is to\n", + "1. Write an abstract for your project\n", + "\n", + "2. Write an introduction\n", + "\n", + "3. Include references\n", + "\n", + "A short feedback to the this exercise will be available before the project deadline. And you can reuse these elements in your final report." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 41110a69c..ec34ece8c 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -37,11 +37,11 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d
  • Plan for week 39
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm
  • Revisiting our Logistic Regression case
  • The equations to solve
  • Solving using Newton-Raphson's method
    • @@ -460,7 +460,7 @@ MathJax.Hub.Config({ -->
    - © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index bdbae8e52..828525e7b 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -190,7 +190,7 @@ MathJax.Hub.Config({
    - © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    @@ -198,29 +198,7 @@ MathJax.Hub.Config({

    Plan for week 39

    -Material for the active learning sessions on Tuesday and Wednesday -

    -

      - -

    • Discussions on how to structure your report for the first project
      • - -

    • Exercise for week 39 on how to write the abstract and the introduction of the report and how to include references.
      • - -

    • Work on project 1, in particular resampling methods like cross-validation and bootstrap. For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11.
      • - -

    • Video on how to write scientific reports recorded during one of the lab sessions
      • -
    -

    -

    These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.

    - -
    - - -
    -Material for the lecture on Thursday September 28 +Material for the lecture on Monday September 23

      @@ -248,11 +226,33 @@ MathJax.Hub.Config({
    +
    +Material for the active learning sessions on Tuesday and Wednesday +

    +

      + +

    • Discussions on how to structure your report for the first project
      • + +

    • Exercise for week 39 on how to write the abstract and the introduction of the report and how to include references.
      • + +

    • Work on project 1, in particular resampling methods like cross-validation and bootstrap. For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11.
      • + +

    • Video on how to write scientific reports recorded during one of the lab sessions
      • +
    +

    +

    These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.

    + +
    + +
    -

    Optimization, the central part of any Machine Learning algortithm

    +

    Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm

    The first few slides here are a repetition from last week.

    diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index 99854828b..442a14b36 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -64,11 +64,11 @@ div.toc p,a {









    -

    Optimization, the central part of any Machine Learning algortithm

    +

    Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm

    The first few slides here are a repetition from last week.

    @@ -3815,7 +3815,7 @@ derivative_fn = grad(sum_logistic)
    - © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index 1893f246a..3a6b19766 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -141,11 +141,11 @@ div.toc p,a {









    -

    Optimization, the central part of any Machine Learning algortithm

    +

    Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm

    The first few slides here are a repetition from last week.

    @@ -3892,7 +3892,7 @@ derivative_fn = grad(sum_logistic)
    - © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index 38807a1b581c8b56cee340a00e4d6892a8105906..77f92f1e8699f9e294268d390a09d3d12b8ff83f 100644 GIT binary patch literal 193 zcmV;y06za8iwFQ<uHia9|^nxw_0U>7a~5igL^)W+JRCMnw6 z+Xv`MaZ^OdxA_@n7-kOHdb7(ScXz>J5JCy1FlL&tDN#Jv6O0*PnxrvFFrx?%CXA2( z&~hiebk=b*oYGWhp{!8v=7zDd{IF+!1)ljQj+L~q-S@823Y2yt*SdzA5H(pvvh7t4 vg%);fff3hE8UbAOz>7jUsTIG3tZ~s5Qe|00;m85xrQ> literal 191 zcmV;w06_mAiwFRmlGia9|^+N9!AunQN0h!;rd)JAPmlN9ak z?GNZmaZ^Odw|RtlgqeM|T5q$+-CZynLP*LO47o`8m?WOwBT6|?#)lG5`vRc}XK?^z zzLj2D>xAi7sjCwzi|Spkua)J8f2LF5nSbI?Nei1@=PIp0X@|MiXSgArlVv2EPUTQ& tw8IDtUR!Apgz7*Pg>+IYaS0ovk4Drs3V(f!=Xsv@wFlVut4{z3003ENR?z?e diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 20f5651c2..b205078f7 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "428bf751", - "metadata": {}, + "id": "c9994a5e", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "1a0a75ed", - "metadata": {}, + "id": "1b9ec753", + "metadata": { + "editable": true + }, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -23,27 +27,14 @@ }, { "cell_type": "markdown", - "id": "333f3063", - "metadata": {}, + "id": "0bfb8a59", + "metadata": { + "editable": true + }, "source": [ "## Plan for week 39\n", "\n", - "**Material for the active learning sessions on Tuesday and Wednesday.**\n", - "\n", - " * Discussions on how to structure your report for the first project\n", - "\n", - " * Exercise for week 39 on how to write the abstract and the introduction of the report and how to include references. \n", - "\n", - " * Work on project 1, in particular resampling methods like cross-validation and bootstrap. **For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11**.\n", - "\n", - " * [Video on how to write scientific reports recorded during one of the lab sessions](https://youtu.be/tVW1ZDmZnwM)\n", - "\n", - "These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. \n", - " * A general guideline can be found at .\n", - "\n", - " \n", - "\n", - "**Material for the lecture on Thursday September 28.**\n", + "**Material for the lecture on Monday September 23.**\n", "\n", " * Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent \n", "\n", @@ -63,15 +54,32 @@ "\n", " * [Video on stochastic gradient descent](https://www.youtube.com/watch?v=vMh0zPT0tLI)\n", "\n", + "**Material for the active learning sessions on Tuesday and Wednesday.**\n", + "\n", + " * Discussions on how to structure your report for the first project\n", + "\n", + " * Exercise for week 39 on how to write the abstract and the introduction of the report and how to include references. \n", + "\n", + " * Work on project 1, in particular resampling methods like cross-validation and bootstrap. **For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11**.\n", + "\n", + " * [Video on how to write scientific reports recorded during one of the lab sessions](https://youtu.be/tVW1ZDmZnwM)\n", + "\n", + "These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. \n", + " * A general guideline can be found at .\n", + "\n", + " \n", + "\n", "" ] }, { "cell_type": "markdown", - "id": "3ee03ecd", - "metadata": {}, + "id": "d073df4a", + "metadata": { + "editable": true + }, "source": [ - "## Optimization, the central part of any Machine Learning algortithm\n", + "## Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm\n", "\n", "The first few slides here are a repetition from last week. \n", "\n", @@ -87,8 +95,10 @@ }, { "cell_type": "markdown", - "id": "148ec577", - "metadata": {}, + "id": "9089187f", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -102,8 +112,10 @@ }, { "cell_type": "markdown", - "id": "e6e5e661", - "metadata": {}, + "id": "52ca6f0c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -115,16 +127,20 @@ }, { "cell_type": "markdown", - "id": "4a81fe9d", - "metadata": {}, + "id": "71aafbc2", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] }, { "cell_type": "markdown", - "id": "ac94af95", - "metadata": {}, + "id": "5ba1be95", + "metadata": { + "editable": true + }, "source": [ "## The equations to solve\n", "\n", @@ -137,8 +153,10 @@ }, { "cell_type": "markdown", - "id": "5bfc3f18", - "metadata": {}, + "id": "cb481897", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -147,8 +165,10 @@ }, { "cell_type": "markdown", - "id": "2437c71a", - "metadata": {}, + "id": "9f81d01b", + "metadata": { + "editable": true + }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -156,8 +176,10 @@ }, { "cell_type": "markdown", - "id": "a5d4163c", - "metadata": {}, + "id": "cb27812e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -166,16 +188,20 @@ }, { "cell_type": "markdown", - "id": "cca433f5", - "metadata": {}, + "id": "6c90f905", + "metadata": { + "editable": true + }, "source": [ "This defines what is called the Hessian matrix." ] }, { "cell_type": "markdown", - "id": "c9ba0136", - "metadata": {}, + "id": "4943a918", + "metadata": { + "editable": true + }, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -186,8 +212,10 @@ }, { "cell_type": "markdown", - "id": "3a6a4c55", - "metadata": {}, + "id": "726bbf45", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -196,16 +224,20 @@ }, { "cell_type": "markdown", - "id": "f71ca135", - "metadata": {}, + "id": "69c95a4c", + "metadata": { + "editable": true + }, "source": [ "or in matrix form as" ] }, { "cell_type": "markdown", - "id": "942a71da", - "metadata": {}, + "id": "2169a0e6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -214,8 +246,10 @@ }, { "cell_type": "markdown", - "id": "3d3ddaff", - "metadata": {}, + "id": "cd876d8a", + "metadata": { + "editable": true + }, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -224,8 +258,10 @@ }, { "cell_type": "markdown", - "id": "3dfb626a", - "metadata": {}, + "id": "57001cd6", + "metadata": { + "editable": true + }, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -242,8 +278,10 @@ }, { "cell_type": "markdown", - "id": "7a77c9ce", - "metadata": {}, + "id": "b9baa74e", + "metadata": { + "editable": true + }, "source": [ "## The equations\n", "\n", @@ -256,8 +294,10 @@ }, { "cell_type": "markdown", - "id": "d094140b", - "metadata": {}, + "id": "03f9dd30", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -270,8 +310,10 @@ }, { "cell_type": "markdown", - "id": "9c04f928", - "metadata": {}, + "id": "5a3865b1", + "metadata": { + "editable": true + }, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -279,8 +321,10 @@ }, { "cell_type": "markdown", - "id": "d29b9b63", - "metadata": {}, + "id": "bd8dabb8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -289,16 +333,20 @@ }, { "cell_type": "markdown", - "id": "d3e508b5", - "metadata": {}, + "id": "d5cd566d", + "metadata": { + "editable": true + }, "source": [ "yielding" ] }, { "cell_type": "markdown", - "id": "fd5bbc31", - "metadata": {}, + "id": "c163bb90", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -307,16 +355,20 @@ }, { "cell_type": "markdown", - "id": "7e255afd", - "metadata": {}, + "id": "d6e18f97", + "metadata": { + "editable": true + }, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] }, { "cell_type": "markdown", - "id": "62d43c1c", - "metadata": {}, + "id": "dd859e9d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -325,8 +377,10 @@ }, { "cell_type": "markdown", - "id": "0b23438c", - "metadata": {}, + "id": "fe53f040", + "metadata": { + "editable": true + }, "source": [ "## Simple geometric interpretation\n", "\n", @@ -345,8 +399,10 @@ }, { "cell_type": "markdown", - "id": "d7587ee4", - "metadata": {}, + "id": "2a212547", + "metadata": { + "editable": true + }, "source": [ "## Extending to more than one variable\n", "\n", @@ -356,8 +412,10 @@ }, { "cell_type": "markdown", - "id": "3736c3ae", - "metadata": {}, + "id": "5412c7d7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -367,16 +425,20 @@ }, { "cell_type": "markdown", - "id": "e9cf6fb6", - "metadata": {}, + "id": "334c83a4", + "metadata": { + "editable": true + }, "source": [ "which we Taylor expand to obtain" ] }, { "cell_type": "markdown", - "id": "9c8e6fc9", - "metadata": {}, + "id": "5a33d078", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -391,16 +453,20 @@ }, { "cell_type": "markdown", - "id": "2059ae9d", - "metadata": {}, + "id": "2de00890", + "metadata": { + "editable": true + }, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { "cell_type": "markdown", - "id": "64c640e4", - "metadata": {}, + "id": "27f71910", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -412,16 +478,20 @@ }, { "cell_type": "markdown", - "id": "647d6bd2", - "metadata": {}, + "id": "39a94110", + "metadata": { + "editable": true + }, "source": [ "we can rephrase Newton's method as" ] }, { "cell_type": "markdown", - "id": "24ff572b", - "metadata": {}, + "id": "781ab586", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -432,16 +502,20 @@ }, { "cell_type": "markdown", - "id": "521c56f5", - "metadata": {}, + "id": "4945c6b7", + "metadata": { + "editable": true + }, "source": [ "where we have defined" ] }, { "cell_type": "markdown", - "id": "4508944b", - "metadata": {}, + "id": "e02fabcd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -452,8 +526,10 @@ }, { "cell_type": "markdown", - "id": "69b72003", - "metadata": {}, + "id": "96a2628b", + "metadata": { + "editable": true + }, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -465,8 +541,10 @@ }, { "cell_type": "markdown", - "id": "dd11ba71", - "metadata": {}, + "id": "8061949d", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent\n", "\n", @@ -480,8 +558,10 @@ }, { "cell_type": "markdown", - "id": "ccb5e1a6", - "metadata": {}, + "id": "5c362006", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -490,8 +570,10 @@ }, { "cell_type": "markdown", - "id": "6952b928", - "metadata": {}, + "id": "cfff5c9a", + "metadata": { + "editable": true + }, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -502,8 +584,10 @@ }, { "cell_type": "markdown", - "id": "f0c5476d", - "metadata": {}, + "id": "e75a4e56", + "metadata": { + "editable": true + }, "source": [ "## More on Steepest descent\n", "\n", @@ -515,8 +599,10 @@ }, { "cell_type": "markdown", - "id": "4aeb7465", - "metadata": {}, + "id": "28d136a8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -525,8 +611,10 @@ }, { "cell_type": "markdown", - "id": "337ccfd3", - "metadata": {}, + "id": "9a7175e0", + "metadata": { + "editable": true + }, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -534,8 +622,10 @@ }, { "cell_type": "markdown", - "id": "7540286d", - "metadata": {}, + "id": "87c0c324", + "metadata": { + "editable": true + }, "source": [ "## The ideal\n", "\n", @@ -560,8 +650,10 @@ }, { "cell_type": "markdown", - "id": "b7ce5820", - "metadata": {}, + "id": "3e48df39", + "metadata": { + "editable": true + }, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -580,8 +672,10 @@ }, { "cell_type": "markdown", - "id": "0a557da3", - "metadata": {}, + "id": "ca63b654", + "metadata": { + "editable": true + }, "source": [ "## Convex functions\n", "\n", @@ -600,8 +694,10 @@ }, { "cell_type": "markdown", - "id": "bdbf8a07", - "metadata": {}, + "id": "84f4f11c", + "metadata": { + "editable": true + }, "source": [ "## Convex function\n", "\n", @@ -619,8 +715,10 @@ }, { "cell_type": "markdown", - "id": "e02d1dac", - "metadata": {}, + "id": "cbe548e4", + "metadata": { + "editable": true + }, "source": [ "## Conditions on convex functions\n", "\n", @@ -656,8 +754,10 @@ }, { "cell_type": "markdown", - "id": "d9a72971", - "metadata": {}, + "id": "67ee6dc6", + "metadata": { + "editable": true + }, "source": [ "## More on convex functions\n", "\n", @@ -682,8 +782,10 @@ }, { "cell_type": "markdown", - "id": "c3e744c2", - "metadata": {}, + "id": "d2691276", + "metadata": { + "editable": true + }, "source": [ "## Some simple problems\n", "\n", @@ -710,8 +812,10 @@ }, { "cell_type": "markdown", - "id": "2732cf8b", - "metadata": {}, + "id": "17919335", + "metadata": { + "editable": true + }, "source": [ "## Standard steepest descent\n", "\n", @@ -728,8 +832,10 @@ }, { "cell_type": "markdown", - "id": "1b14ba6d", - "metadata": {}, + "id": "fa94a9e1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -738,16 +844,20 @@ }, { "cell_type": "markdown", - "id": "5f271cd8", - "metadata": {}, + "id": "5abb2d3c", + "metadata": { + "editable": true + }, "source": [ "In the iterative process we end up with a problem like" ] }, { "cell_type": "markdown", - "id": "eb0ce7c8", - "metadata": {}, + "id": "b365b48f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -756,8 +866,10 @@ }, { "cell_type": "markdown", - "id": "3482f635", - "metadata": {}, + "id": "4e3742ba", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -766,8 +878,10 @@ }, { "cell_type": "markdown", - "id": "0c88f8d2", - "metadata": {}, + "id": "58ed86be", + "metadata": { + "editable": true + }, "source": [ "## Gradient method\n", "\n", @@ -776,8 +890,10 @@ }, { "cell_type": "markdown", - "id": "fe152e73", - "metadata": {}, + "id": "2fd318b5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -786,8 +902,10 @@ }, { "cell_type": "markdown", - "id": "740c5860", - "metadata": {}, + "id": "69f6cc1b", + "metadata": { + "editable": true + }, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -795,8 +913,10 @@ }, { "cell_type": "markdown", - "id": "477da242", - "metadata": {}, + "id": "2ba7b23b", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "\n", @@ -806,8 +926,10 @@ }, { "cell_type": "markdown", - "id": "7e2169e6", - "metadata": {}, + "id": "bd6d7d21", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -816,16 +938,20 @@ }, { "cell_type": "markdown", - "id": "e513c1c5", - "metadata": {}, + "id": "c64a8e61", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "a4a4f67c", - "metadata": {}, + "id": "66952b59", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -834,16 +960,20 @@ }, { "cell_type": "markdown", - "id": "96138e83", - "metadata": {}, + "id": "7f56e6d6", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "f843d9f8", - "metadata": {}, + "id": "d5cc98be", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -851,8 +981,10 @@ }, { "cell_type": "markdown", - "id": "5702234f", - "metadata": {}, + "id": "6885bb8e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -861,8 +993,10 @@ }, { "cell_type": "markdown", - "id": "682a415f", - "metadata": {}, + "id": "cd913484", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -871,8 +1005,10 @@ }, { "cell_type": "markdown", - "id": "302ac54a", - "metadata": {}, + "id": "a1a2884d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -881,8 +1017,10 @@ }, { "cell_type": "markdown", - "id": "dd2cbeb1", - "metadata": {}, + "id": "ca323749", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -890,8 +1028,10 @@ }, { "cell_type": "markdown", - "id": "f280370e", - "metadata": {}, + "id": "a4f88c2f", + "metadata": { + "editable": true + }, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -899,8 +1039,10 @@ }, { "cell_type": "markdown", - "id": "9702a162", - "metadata": {}, + "id": "5d7ee548", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -909,16 +1051,20 @@ }, { "cell_type": "markdown", - "id": "3a1de3f0", - "metadata": {}, + "id": "ccb21024", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "4fa494ea", - "metadata": {}, + "id": "e5ce8796", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -927,16 +1073,20 @@ }, { "cell_type": "markdown", - "id": "a4f12308", - "metadata": {}, + "id": "db766e68", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "df770c35", - "metadata": {}, + "id": "06851dbc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -945,16 +1095,20 @@ }, { "cell_type": "markdown", - "id": "b1a30174", - "metadata": {}, + "id": "170f1238", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "6f816a49", - "metadata": {}, + "id": "bc12a3fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -963,16 +1117,20 @@ }, { "cell_type": "markdown", - "id": "4e5cd41c", - "metadata": {}, + "id": "08509fc2", + "metadata": { + "editable": true + }, "source": [ "leading to the iterative scheme" ] }, { "cell_type": "markdown", - "id": "91f972cb", - "metadata": {}, + "id": "dce4db11", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_{k},\n", @@ -981,8 +1139,10 @@ }, { "cell_type": "markdown", - "id": "97064af3", - "metadata": {}, + "id": "4431d466", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent example" ] @@ -990,8 +1150,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "5af74f1d", - "metadata": {}, + "id": "73a0c80a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -1020,8 +1183,10 @@ }, { "cell_type": "markdown", - "id": "db2d4ba3", - "metadata": {}, + "id": "44d3dd5c", + "metadata": { + "editable": true + }, "source": [ "And then as countor plot" ] @@ -1029,8 +1194,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "dbba72eb", - "metadata": {}, + "id": "4cf3e064", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1040,8 +1208,10 @@ }, { "cell_type": "markdown", - "id": "8290c8f1", - "metadata": {}, + "id": "3b60fca7", + "metadata": { + "editable": true + }, "source": [ "Find guesses" ] @@ -1049,8 +1219,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "ce55e78a", - "metadata": {}, + "id": "376f7666", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1059,8 +1232,10 @@ }, { "cell_type": "markdown", - "id": "0fa3681b", - "metadata": {}, + "id": "5d6fd366", + "metadata": { + "editable": true + }, "source": [ "Run it!" ] @@ -1068,8 +1243,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "a5aee074", - "metadata": {}, + "id": "302f38b9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1083,8 +1261,10 @@ }, { "cell_type": "markdown", - "id": "8f8ed4d2", - "metadata": {}, + "id": "d655b91c", + "metadata": { + "editable": true + }, "source": [ "What happened?" ] @@ -1092,8 +1272,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "30855606", - "metadata": {}, + "id": "e959b3c6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1104,16 +1287,20 @@ }, { "cell_type": "markdown", - "id": "fcd5a0c8", - "metadata": {}, + "id": "d6b5e06a", + "metadata": { + "editable": true + }, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] }, { "cell_type": "markdown", - "id": "acdc3658", - "metadata": {}, + "id": "c6e9d003", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1124,8 +1311,10 @@ }, { "cell_type": "markdown", - "id": "7e07632f", - "metadata": {}, + "id": "fc3f89ea", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1134,8 +1323,10 @@ }, { "cell_type": "markdown", - "id": "356d5fe1", - "metadata": {}, + "id": "f24b56b6", + "metadata": { + "editable": true + }, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1143,8 +1334,10 @@ }, { "cell_type": "markdown", - "id": "2033af61", - "metadata": {}, + "id": "57cca740", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1153,8 +1346,10 @@ }, { "cell_type": "markdown", - "id": "0a85c783", - "metadata": {}, + "id": "e962dfeb", + "metadata": { + "editable": true + }, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1162,8 +1357,10 @@ }, { "cell_type": "markdown", - "id": "6c19d77d", - "metadata": {}, + "id": "4adde7eb", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1171,8 +1368,10 @@ }, { "cell_type": "markdown", - "id": "f42364fb", - "metadata": {}, + "id": "f66487e4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1181,16 +1380,20 @@ }, { "cell_type": "markdown", - "id": "c841e7d3", - "metadata": {}, + "id": "78361f9d", + "metadata": { + "editable": true + }, "source": [ "which is zero unless $i=j$." ] }, { "cell_type": "markdown", - "id": "297492ba", - "metadata": {}, + "id": "b8ce6414", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1199,8 +1402,10 @@ }, { "cell_type": "markdown", - "id": "4963a2d8", - "metadata": {}, + "id": "3a43e97b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1209,8 +1414,10 @@ }, { "cell_type": "markdown", - "id": "4a39d88b", - "metadata": {}, + "id": "0eee3d63", + "metadata": { + "editable": true + }, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1219,8 +1426,10 @@ }, { "cell_type": "markdown", - "id": "83a86148", - "metadata": {}, + "id": "44163294", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1229,8 +1438,10 @@ }, { "cell_type": "markdown", - "id": "c691f06b", - "metadata": {}, + "id": "b05293c0", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1238,8 +1449,10 @@ }, { "cell_type": "markdown", - "id": "d4df90e8", - "metadata": {}, + "id": "1e20ca51", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1248,16 +1461,20 @@ }, { "cell_type": "markdown", - "id": "bf3217ad", - "metadata": {}, + "id": "053f3d59", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] }, { "cell_type": "markdown", - "id": "b43b4b20", - "metadata": {}, + "id": "50fe198d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1266,16 +1483,20 @@ }, { "cell_type": "markdown", - "id": "d82bb554", - "metadata": {}, + "id": "d8d565b2", + "metadata": { + "editable": true + }, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] }, { "cell_type": "markdown", - "id": "dde5ed03", - "metadata": {}, + "id": "3c4a52ca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1284,8 +1505,10 @@ }, { "cell_type": "markdown", - "id": "65ecffe4", - "metadata": {}, + "id": "6cb00316", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1302,8 +1525,10 @@ }, { "cell_type": "markdown", - "id": "65a1b4c0", - "metadata": {}, + "id": "d7954a8c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1312,16 +1537,20 @@ }, { "cell_type": "markdown", - "id": "41dec06b", - "metadata": {}, + "id": "f4e00bac", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "7d1da8a1", - "metadata": {}, + "id": "fc2e6579", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1330,16 +1559,20 @@ }, { "cell_type": "markdown", - "id": "c4850b28", - "metadata": {}, + "id": "ebaa3afb", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "ff91db7e", - "metadata": {}, + "id": "f36132b6", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1347,8 +1580,10 @@ }, { "cell_type": "markdown", - "id": "a8ab3c6e", - "metadata": {}, + "id": "343ac54a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1357,8 +1592,10 @@ }, { "cell_type": "markdown", - "id": "e30ea38e", - "metadata": {}, + "id": "a8252550", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1367,8 +1604,10 @@ }, { "cell_type": "markdown", - "id": "ede52cdd", - "metadata": {}, + "id": "4f369be7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1377,8 +1616,10 @@ }, { "cell_type": "markdown", - "id": "3cdf6ffd", - "metadata": {}, + "id": "90b2c7ab", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1388,8 +1629,10 @@ }, { "cell_type": "markdown", - "id": "d66ac756", - "metadata": {}, + "id": "0c986a6a", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1397,8 +1640,10 @@ }, { "cell_type": "markdown", - "id": "59b7a9f5", - "metadata": {}, + "id": "2a6d835c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1407,8 +1652,10 @@ }, { "cell_type": "markdown", - "id": "769980be", - "metadata": {}, + "id": "0d5ee6c7", + "metadata": { + "editable": true + }, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1421,8 +1668,10 @@ }, { "cell_type": "markdown", - "id": "3ae7691f", - "metadata": {}, + "id": "795cf2b0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1431,8 +1680,10 @@ }, { "cell_type": "markdown", - "id": "9cf530dc", - "metadata": {}, + "id": "152a8331", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1440,8 +1691,10 @@ }, { "cell_type": "markdown", - "id": "6398a5d7", - "metadata": {}, + "id": "f0e4ce53", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1450,16 +1703,20 @@ }, { "cell_type": "markdown", - "id": "b64c8282", - "metadata": {}, + "id": "5f95488a", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "4b9bcf21", - "metadata": {}, + "id": "dd4b1562", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1468,16 +1725,20 @@ }, { "cell_type": "markdown", - "id": "6d15e4b9", - "metadata": {}, + "id": "e637a0a3", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "592ac2b7", - "metadata": {}, + "id": "70feaafd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1486,16 +1747,20 @@ }, { "cell_type": "markdown", - "id": "c54989ac", - "metadata": {}, + "id": "a4bde000", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "87adadc2", - "metadata": {}, + "id": "9c678b79", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1504,8 +1769,10 @@ }, { "cell_type": "markdown", - "id": "00250e67", - "metadata": {}, + "id": "dcee45d4", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our first homework\n", "\n", @@ -1526,8 +1793,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "47a98c7c", - "metadata": {}, + "id": "710fc496", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1536,8 +1806,10 @@ }, { "cell_type": "markdown", - "id": "e3007f3b", - "metadata": {}, + "id": "173e1c77", + "metadata": { + "editable": true + }, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1545,8 +1817,10 @@ }, { "cell_type": "markdown", - "id": "c5b7179d", - "metadata": {}, + "id": "94aa0321", + "metadata": { + "editable": true + }, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1555,16 +1829,20 @@ }, { "cell_type": "markdown", - "id": "8a4d63b4", - "metadata": {}, + "id": "61164cb2", + "metadata": { + "editable": true + }, "source": [ "such that" ] }, { "cell_type": "markdown", - "id": "206c9402", - "metadata": {}, + "id": "51b088c2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1573,8 +1851,10 @@ }, { "cell_type": "markdown", - "id": "939b3b78", - "metadata": {}, + "id": "c113ace2", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent example\n", "\n", @@ -1585,8 +1865,10 @@ }, { "cell_type": "markdown", - "id": "b867af05", - "metadata": {}, + "id": "4ea11bb0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1599,16 +1881,20 @@ }, { "cell_type": "markdown", - "id": "76738c60", - "metadata": {}, + "id": "cd9d1d31", + "metadata": { + "editable": true + }, "source": [ "The cost/loss/risk function is given by (" ] }, { "cell_type": "markdown", - "id": "c01273f5", - "metadata": {}, + "id": "4e1f9a20", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1617,16 +1903,20 @@ }, { "cell_type": "markdown", - "id": "5d680787", - "metadata": {}, + "id": "1cd7bb21", + "metadata": { + "editable": true + }, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] }, { "cell_type": "markdown", - "id": "2ef9ff3b", - "metadata": {}, + "id": "e1968965", + "metadata": { + "editable": true + }, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -1635,8 +1925,10 @@ }, { "cell_type": "markdown", - "id": "e5a81fba", - "metadata": {}, + "id": "695fa7b5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1647,16 +1939,20 @@ }, { "cell_type": "markdown", - "id": "b7298ace", - "metadata": {}, + "id": "62268e92", + "metadata": { + "editable": true + }, "source": [ "where $X$ is the design matrix defined above." ] }, { "cell_type": "markdown", - "id": "64cfb75f", - "metadata": {}, + "id": "1362e220", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -1664,8 +1960,10 @@ }, { "cell_type": "markdown", - "id": "99503e16", - "metadata": {}, + "id": "015cd1c3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1677,16 +1975,20 @@ }, { "cell_type": "markdown", - "id": "4a567780", - "metadata": {}, + "id": "0f5d535c", + "metadata": { + "editable": true + }, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] }, { "cell_type": "markdown", - "id": "22c576da", - "metadata": {}, + "id": "ca5e2d3d", + "metadata": { + "editable": true + }, "source": [ "## Simple program\n", "\n", @@ -1695,8 +1997,10 @@ }, { "cell_type": "markdown", - "id": "44a99f62", - "metadata": {}, + "id": "bb43ed51", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1705,8 +2009,10 @@ }, { "cell_type": "markdown", - "id": "7021c749", - "metadata": {}, + "id": "a3a0a6ef", + "metadata": { + "editable": true + }, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -1718,8 +2024,10 @@ }, { "cell_type": "markdown", - "id": "6044e7a8", - "metadata": {}, + "id": "3fd1e6b5", + "metadata": { + "editable": true + }, "source": [ "## Gradient Descent Example\n", "\n", @@ -1729,8 +2037,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "72003ff9", - "metadata": {}, + "id": "da10b23b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1783,8 +2094,10 @@ }, { "cell_type": "markdown", - "id": "01fdfcaf", - "metadata": {}, + "id": "f94b865f", + "metadata": { + "editable": true + }, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -1792,8 +2105,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "d16ddbdc", - "metadata": {}, + "id": "415b067b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -1816,8 +2132,10 @@ }, { "cell_type": "markdown", - "id": "08aaf479", - "metadata": {}, + "id": "032f6c14", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -1826,8 +2144,10 @@ }, { "cell_type": "markdown", - "id": "0aa5045f", - "metadata": {}, + "id": "d34dbdd9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -1836,16 +2156,20 @@ }, { "cell_type": "markdown", - "id": "6474d14b", - "metadata": {}, + "id": "3f9145d7", + "metadata": { + "editable": true + }, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] }, { "cell_type": "markdown", - "id": "9335b39d", - "metadata": {}, + "id": "633b1b24", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1856,16 +2180,20 @@ }, { "cell_type": "markdown", - "id": "0680a59f", - "metadata": {}, + "id": "94d03916", + "metadata": { + "editable": true + }, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] }, { "cell_type": "markdown", - "id": "de5afdeb", - "metadata": {}, + "id": "b0373920", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -1874,8 +2202,10 @@ }, { "cell_type": "markdown", - "id": "0042d7e6", - "metadata": {}, + "id": "3cb79fc3", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -1883,8 +2213,10 @@ }, { "cell_type": "markdown", - "id": "02cf311f", - "metadata": {}, + "id": "0e1b4743", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1896,8 +2228,10 @@ }, { "cell_type": "markdown", - "id": "3dbc50e6", - "metadata": {}, + "id": "985ad420", + "metadata": { + "editable": true + }, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -1908,8 +2242,10 @@ }, { "cell_type": "markdown", - "id": "437e17bc", - "metadata": {}, + "id": "ef940565", + "metadata": { + "editable": true + }, "source": [ "## Program example for gradient descent with Ridge Regression" ] @@ -1917,8 +2253,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "f11ee927", - "metadata": {}, + "id": "0f3bf2d6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from random import random, seed\n", @@ -1975,8 +2314,10 @@ }, { "cell_type": "markdown", - "id": "c06cf31f", - "metadata": {}, + "id": "d9aaf64b", + "metadata": { + "editable": true + }, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -1995,8 +2336,10 @@ }, { "cell_type": "markdown", - "id": "870586cd", - "metadata": {}, + "id": "bdb90484", + "metadata": { + "editable": true + }, "source": [ "## Improving gradient descent with momentum\n", "\n", @@ -2006,8 +2349,11 @@ { "cell_type": "code", "execution_count": 10, - "id": "24517bb5", - "metadata": {}, + "id": "2ad4f34d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from numpy import asarray\n", @@ -2069,8 +2415,10 @@ }, { "cell_type": "markdown", - "id": "57647429", - "metadata": {}, + "id": "21d1e1dd", + "metadata": { + "editable": true + }, "source": [ "## Same code but now with momentum gradient descent" ] @@ -2078,8 +2426,11 @@ { "cell_type": "code", "execution_count": 11, - "id": "b8365c93", - "metadata": {}, + "id": "3e23efef", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from numpy import asarray\n", @@ -2149,8 +2500,10 @@ }, { "cell_type": "markdown", - "id": "3c5b105d", - "metadata": {}, + "id": "8ba5c33f", + "metadata": { + "editable": true + }, "source": [ "## Overview video on Stochastic Gradient Descent\n", "\n", @@ -2159,8 +2512,10 @@ }, { "cell_type": "markdown", - "id": "e78e4fcf", - "metadata": {}, + "id": "4a28ffaa", + "metadata": { + "editable": true + }, "source": [ "## Batches and mini-batches\n", "\n", @@ -2178,8 +2533,10 @@ }, { "cell_type": "markdown", - "id": "9e856c0b", - "metadata": {}, + "id": "4afa3635", + "metadata": { + "editable": true + }, "source": [ "## Stochastic Gradient Descent (SGD)\n", "\n", @@ -2208,8 +2565,10 @@ }, { "cell_type": "markdown", - "id": "8400c2e5", - "metadata": {}, + "id": "58297357", + "metadata": { + "editable": true + }, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2223,8 +2582,10 @@ }, { "cell_type": "markdown", - "id": "d0ceff52", - "metadata": {}, + "id": "ff7efa58", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2234,8 +2595,10 @@ }, { "cell_type": "markdown", - "id": "562ca1d7", - "metadata": {}, + "id": "5060ff9d", + "metadata": { + "editable": true + }, "source": [ "## Computation of gradients\n", "\n", @@ -2245,8 +2608,10 @@ }, { "cell_type": "markdown", - "id": "ffea7df9", - "metadata": {}, + "id": "7a907783", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2256,8 +2621,10 @@ }, { "cell_type": "markdown", - "id": "20f1bd07", - "metadata": {}, + "id": "8e114147", + "metadata": { + "editable": true + }, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2268,8 +2635,10 @@ }, { "cell_type": "markdown", - "id": "4589bb1b", - "metadata": {}, + "id": "f9cbed09", + "metadata": { + "editable": true + }, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2288,8 +2657,10 @@ }, { "cell_type": "markdown", - "id": "0df2146b", - "metadata": {}, + "id": "9211b1ac", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2301,8 +2672,10 @@ }, { "cell_type": "markdown", - "id": "890e6746", - "metadata": {}, + "id": "f13802c2", + "metadata": { + "editable": true + }, "source": [ "## The gradient step\n", "\n", @@ -2311,8 +2684,10 @@ }, { "cell_type": "markdown", - "id": "b6e42059", - "metadata": {}, + "id": "672ab979", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2322,8 +2697,10 @@ }, { "cell_type": "markdown", - "id": "9dd3abbf", - "metadata": {}, + "id": "87a7febb", + "metadata": { + "editable": true + }, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2334,8 +2711,10 @@ }, { "cell_type": "markdown", - "id": "97279f92", - "metadata": {}, + "id": "47b544ca", + "metadata": { + "editable": true + }, "source": [ "## Simple example code" ] @@ -2343,8 +2722,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "cb0c6322", - "metadata": {}, + "id": "26d07242", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2365,8 +2747,10 @@ }, { "cell_type": "markdown", - "id": "c0868aae", - "metadata": {}, + "id": "e32d96f5", + "metadata": { + "editable": true + }, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2379,8 +2763,10 @@ }, { "cell_type": "markdown", - "id": "1e17bb0f", - "metadata": {}, + "id": "fc458e55", + "metadata": { + "editable": true + }, "source": [ "## When do we stop?\n", "\n", @@ -2398,8 +2784,10 @@ }, { "cell_type": "markdown", - "id": "f050ca70", - "metadata": {}, + "id": "66ccb6ed", + "metadata": { + "editable": true + }, "source": [ "## Slightly different approach\n", "\n", @@ -2416,8 +2804,10 @@ }, { "cell_type": "markdown", - "id": "6a900f78", - "metadata": {}, + "id": "3a94b654", + "metadata": { + "editable": true + }, "source": [ "## Time decay rate\n", "\n", @@ -2433,8 +2823,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "1324db42", - "metadata": {}, + "id": "ef97a06b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2465,8 +2858,10 @@ }, { "cell_type": "markdown", - "id": "0c365408", - "metadata": {}, + "id": "274e2be8", + "metadata": { + "editable": true + }, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2476,8 +2871,11 @@ { "cell_type": "code", "execution_count": 14, - "id": "79417e81", - "metadata": {}, + "id": "275e7463", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -2550,8 +2948,10 @@ }, { "cell_type": "markdown", - "id": "dd16fd67", - "metadata": {}, + "id": "3c6c95e2", + "metadata": { + "editable": true + }, "source": [ "## Replace or not\n", "\n", @@ -2563,8 +2963,10 @@ }, { "cell_type": "markdown", - "id": "2bbf7fbd", - "metadata": {}, + "id": "972dae2f", + "metadata": { + "editable": true + }, "source": [ "## Momentum based GD\n", "\n", @@ -2576,8 +2978,10 @@ }, { "cell_type": "markdown", - "id": "d4aa4448", - "metadata": {}, + "id": "44b295c1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", @@ -2586,8 +2990,10 @@ }, { "cell_type": "markdown", - "id": "fdcd258f", - "metadata": {}, + "id": "28835d66", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2602,8 +3008,10 @@ }, { "cell_type": "markdown", - "id": "52ec5bfb", - "metadata": {}, + "id": "1c1b3779", + "metadata": { + "editable": true + }, "source": [ "where we have introduced a momentum parameter $\\gamma$, with\n", "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", @@ -2619,8 +3027,10 @@ }, { "cell_type": "markdown", - "id": "38004062", - "metadata": {}, + "id": "de200956", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", @@ -2629,16 +3039,20 @@ }, { "cell_type": "markdown", - "id": "9d07c567", - "metadata": {}, + "id": "07f1055e", + "metadata": { + "editable": true + }, "source": [ "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." ] }, { "cell_type": "markdown", - "id": "4dbba8bc", - "metadata": {}, + "id": "c250d166", + "metadata": { + "editable": true + }, "source": [ "## More on momentum based approaches\n", "\n", @@ -2651,8 +3065,10 @@ }, { "cell_type": "markdown", - "id": "dab76529", - "metadata": {}, + "id": "b9fb53fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", @@ -2661,16 +3077,20 @@ }, { "cell_type": "markdown", - "id": "6e049d15", - "metadata": {}, + "id": "7d119ae8", + "metadata": { + "editable": true + }, "source": [ "We can discretize this equation in the usual way to get" ] }, { "cell_type": "markdown", - "id": "5079f465", - "metadata": {}, + "id": "e0fa7dc0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", @@ -2679,16 +3099,20 @@ }, { "cell_type": "markdown", - "id": "51c2ed45", - "metadata": {}, + "id": "83df3160", + "metadata": { + "editable": true + }, "source": [ "Rearranging this equation, we can rewrite this as" ] }, { "cell_type": "markdown", - "id": "7e8f7b16", - "metadata": {}, + "id": "23eb9411", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", @@ -2697,8 +3121,10 @@ }, { "cell_type": "markdown", - "id": "ae0505aa", - "metadata": {}, + "id": "004a68b0", + "metadata": { + "editable": true + }, "source": [ "## Momentum parameter\n", "\n", @@ -2711,8 +3137,10 @@ }, { "cell_type": "markdown", - "id": "9e7f520b", - "metadata": {}, + "id": "717b824a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", @@ -2721,8 +3149,10 @@ }, { "cell_type": "markdown", - "id": "5c0aa1f6", - "metadata": {}, + "id": "be1a7d30", + "metadata": { + "editable": true + }, "source": [ "Thus, as the name suggests, the momentum parameter is proportional to\n", "the mass of the particle and effectively provides inertia.\n", @@ -2752,8 +3182,10 @@ }, { "cell_type": "markdown", - "id": "991c2c15", - "metadata": {}, + "id": "9cd1514e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", @@ -2762,8 +3194,10 @@ }, { "cell_type": "markdown", - "id": "c643afb5", - "metadata": {}, + "id": "0f746bb9", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2778,16 +3212,20 @@ }, { "cell_type": "markdown", - "id": "ac4a060d", - "metadata": {}, + "id": "2e54104b", + "metadata": { + "editable": true + }, "source": [ "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$." ] }, { "cell_type": "markdown", - "id": "37584d4d", - "metadata": {}, + "id": "6b8d5f36", + "metadata": { + "editable": true + }, "source": [ "## Second moment of the gradient\n", "\n", @@ -2815,8 +3253,10 @@ }, { "cell_type": "markdown", - "id": "0e9c907f", - "metadata": {}, + "id": "0ce9a395", + "metadata": { + "editable": true + }, "source": [ "## RMS prop\n", "\n", @@ -2828,8 +3268,10 @@ }, { "cell_type": "markdown", - "id": "cb4567f1", - "metadata": {}, + "id": "512002ce", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2844,8 +3286,10 @@ }, { "cell_type": "markdown", - "id": "71805d3d", - "metadata": {}, + "id": "0ce18424", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", @@ -2854,8 +3298,10 @@ }, { "cell_type": "markdown", - "id": "09794996", - "metadata": {}, + "id": "9f1b1dda", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", @@ -2864,8 +3310,10 @@ }, { "cell_type": "markdown", - "id": "aeb48f66", - "metadata": {}, + "id": "8a053ddc", + "metadata": { + "editable": true + }, "source": [ "where $\\beta$ controls the averaging time of the second moment and is\n", "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", @@ -2880,8 +3328,10 @@ }, { "cell_type": "markdown", - "id": "68e08134", - "metadata": {}, + "id": "68e9ea24", + "metadata": { + "editable": true + }, "source": [ "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", "\n", @@ -2907,8 +3357,10 @@ }, { "cell_type": "markdown", - "id": "69308397", - "metadata": {}, + "id": "f7e99df8", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2923,8 +3375,10 @@ }, { "cell_type": "markdown", - "id": "d23ab794", - "metadata": {}, + "id": "4addc25e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", @@ -2933,8 +3387,10 @@ }, { "cell_type": "markdown", - "id": "c4cef70b", - "metadata": {}, + "id": "441efafd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", @@ -2943,8 +3399,10 @@ }, { "cell_type": "markdown", - "id": "6aebd1b5", - "metadata": {}, + "id": "f2fc97aa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", @@ -2953,8 +3411,10 @@ }, { "cell_type": "markdown", - "id": "c43fe267", - "metadata": {}, + "id": "bff7525d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", @@ -2963,8 +3423,10 @@ }, { "cell_type": "markdown", - "id": "9ae56692", - "metadata": {}, + "id": "f7821a79", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", @@ -2973,8 +3435,10 @@ }, { "cell_type": "markdown", - "id": "784ba00e", - "metadata": {}, + "id": "872174ea", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2988,8 +3452,10 @@ }, { "cell_type": "markdown", - "id": "a187dfb2", - "metadata": {}, + "id": "e9314f50", + "metadata": { + "editable": true + }, "source": [ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", @@ -3005,8 +3471,10 @@ }, { "cell_type": "markdown", - "id": "832cc99c", - "metadata": {}, + "id": "76f7f4f9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", @@ -3015,8 +3483,10 @@ }, { "cell_type": "markdown", - "id": "b1a24342", - "metadata": {}, + "id": "6cebe7fb", + "metadata": { + "editable": true + }, "source": [ "## Algorithms and codes for Adagrad, RMSprop and Adam\n", "\n", @@ -3027,8 +3497,10 @@ }, { "cell_type": "markdown", - "id": "8a97509f", - "metadata": {}, + "id": "86db9944", + "metadata": { + "editable": true + }, "source": [ "## Practical tips\n", "\n", @@ -3045,8 +3517,10 @@ }, { "cell_type": "markdown", - "id": "631a2aa8", - "metadata": {}, + "id": "d184a90d", + "metadata": { + "editable": true + }, "source": [ "## Automatic differentiation\n", "\n", @@ -3081,8 +3555,10 @@ }, { "cell_type": "markdown", - "id": "472b23f2", - "metadata": {}, + "id": "adace522", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x) = \\sin\\left(2\\pi x + x^2\\right)\n", @@ -3091,16 +3567,20 @@ }, { "cell_type": "markdown", - "id": "1c91c90f", - "metadata": {}, + "id": "abb76f90", + "metadata": { + "editable": true + }, "source": [ "which has the following derivative" ] }, { "cell_type": "markdown", - "id": "a85c6aab", - "metadata": {}, + "id": "7469bfb2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n", @@ -3109,8 +3589,10 @@ }, { "cell_type": "markdown", - "id": "89a0bdbb", - "metadata": {}, + "id": "d1b73548", + "metadata": { + "editable": true + }, "source": [ "Using **autograd** we have" ] @@ -3118,8 +3600,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "6fe48a50", - "metadata": {}, + "id": "188a23de", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3160,8 +3645,10 @@ }, { "cell_type": "markdown", - "id": "cab7d753", - "metadata": {}, + "id": "dbdcc55d", + "metadata": { + "editable": true + }, "source": [ "## Using autograd\n", "\n", @@ -3175,8 +3662,11 @@ { "cell_type": "code", "execution_count": 16, - "id": "ca4d6b32", - "metadata": {}, + "id": "e576c5b5", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3200,8 +3690,10 @@ }, { "cell_type": "markdown", - "id": "4a748513", - "metadata": {}, + "id": "410b0fbd", + "metadata": { + "editable": true + }, "source": [ "## Autograd with more complicated functions\n", "\n", @@ -3213,8 +3705,11 @@ { "cell_type": "code", "execution_count": 17, - "id": "f235cd43", - "metadata": {}, + "id": "a0393ee6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3254,16 +3749,20 @@ }, { "cell_type": "markdown", - "id": "5d8df033", - "metadata": {}, + "id": "446967ee", + "metadata": { + "editable": true + }, "source": [ "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable." ] }, { "cell_type": "markdown", - "id": "2f7de144", - "metadata": {}, + "id": "dde4398c", + "metadata": { + "editable": true + }, "source": [ "## More complicated functions using the elements of their arguments directly" ] @@ -3271,8 +3770,11 @@ { "cell_type": "code", "execution_count": 18, - "id": "a2ed8fd6", - "metadata": {}, + "id": "a75cb17e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3296,8 +3798,10 @@ }, { "cell_type": "markdown", - "id": "1f633736", - "metadata": {}, + "id": "180bf59e", + "metadata": { + "editable": true + }, "source": [ "Note that in this case, when sending an array as input argument, the\n", "output from Autograd is another array. This is the true gradient of\n", @@ -3309,8 +3813,10 @@ }, { "cell_type": "markdown", - "id": "a73225d4", - "metadata": {}, + "id": "377739a9", + "metadata": { + "editable": true + }, "source": [ "## Functions using mathematical functions from Numpy" ] @@ -3318,8 +3824,11 @@ { "cell_type": "code", "execution_count": 19, - "id": "39059d20", - "metadata": {}, + "id": "e128f739", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3343,8 +3852,10 @@ }, { "cell_type": "markdown", - "id": "d147b69e", - "metadata": {}, + "id": "860b1abf", + "metadata": { + "editable": true + }, "source": [ "## More autograd" ] @@ -3352,8 +3863,11 @@ { "cell_type": "code", "execution_count": 20, - "id": "90a2d143", - "metadata": {}, + "id": "8cd2c18c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3374,8 +3888,10 @@ }, { "cell_type": "markdown", - "id": "6655faec", - "metadata": {}, + "id": "55a746a6", + "metadata": { + "editable": true + }, "source": [ "## And with loops" ] @@ -3383,8 +3899,11 @@ { "cell_type": "code", "execution_count": 21, - "id": "639ea2a9", - "metadata": {}, + "id": "0b27c49c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3416,8 +3935,11 @@ { "cell_type": "code", "execution_count": 22, - "id": "a53c5633", - "metadata": {}, + "id": "655c9292", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3433,8 +3955,10 @@ }, { "cell_type": "markdown", - "id": "be5e41d4", - "metadata": {}, + "id": "29ce4a15", + "metadata": { + "editable": true + }, "source": [ "## Using recursion" ] @@ -3442,8 +3966,11 @@ { "cell_type": "code", "execution_count": 23, - "id": "786e19d0", - "metadata": {}, + "id": "c79d9983", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3477,16 +4004,20 @@ }, { "cell_type": "markdown", - "id": "114e7e25", - "metadata": {}, + "id": "76a19059", + "metadata": { + "editable": true + }, "source": [ "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input." ] }, { "cell_type": "markdown", - "id": "24c8ffa6", - "metadata": {}, + "id": "51f6d8ad", + "metadata": { + "editable": true + }, "source": [ "## Unsupported functions\n", "Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n", @@ -3497,8 +4028,11 @@ { "cell_type": "code", "execution_count": 24, - "id": "59f521ac", - "metadata": {}, + "id": "3d9492af", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3516,16 +4050,20 @@ }, { "cell_type": "markdown", - "id": "686c34bb", - "metadata": {}, + "id": "960858b4", + "metadata": { + "editable": true + }, "source": [ "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible." ] }, { "cell_type": "markdown", - "id": "9b4cc4f3", - "metadata": {}, + "id": "7b9a8902", + "metadata": { + "editable": true + }, "source": [ "## The syntax a.dot(b) when finding the dot product" ] @@ -3533,8 +4071,11 @@ { "cell_type": "code", "execution_count": 25, - "id": "dea954af", - "metadata": {}, + "id": "cd232c8b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3552,8 +4093,10 @@ }, { "cell_type": "markdown", - "id": "9732d039", - "metadata": {}, + "id": "4cd68dd4", + "metadata": { + "editable": true + }, "source": [ "Here we are told that the 'dot' function does not belong to Autograd's\n", "version of a Numpy array. To overcome this, an alternative syntax\n", @@ -3563,8 +4106,11 @@ { "cell_type": "code", "execution_count": 26, - "id": "f580c6a1", - "metadata": {}, + "id": "7da78e4d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3585,8 +4131,10 @@ }, { "cell_type": "markdown", - "id": "d8714004", - "metadata": {}, + "id": "da924843", + "metadata": { + "editable": true + }, "source": [ "## Recommended to avoid\n", "The documentation recommends to avoid inplace operations such as" @@ -3595,8 +4143,11 @@ { "cell_type": "code", "execution_count": 27, - "id": "e56cbb47", - "metadata": {}, + "id": "95a73352", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "a += b\n", @@ -3607,8 +4158,10 @@ }, { "cell_type": "markdown", - "id": "fac1a7da", - "metadata": {}, + "id": "cf4a75da", + "metadata": { + "editable": true + }, "source": [ "## Using Autograd with OLS\n", "\n", @@ -3620,8 +4173,11 @@ { "cell_type": "code", "execution_count": 28, - "id": "c4c5b9c0", - "metadata": {}, + "id": "105e5771", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients for OLS\n", @@ -3677,8 +4233,10 @@ }, { "cell_type": "markdown", - "id": "b16d7700", - "metadata": {}, + "id": "be689e89", + "metadata": { + "editable": true + }, "source": [ "## Same code but now with momentum gradient descent" ] @@ -3686,8 +4244,11 @@ { "cell_type": "code", "execution_count": 29, - "id": "7453efe5", - "metadata": {}, + "id": "3cd586f0", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients for OLS\n", @@ -3747,8 +4308,10 @@ }, { "cell_type": "markdown", - "id": "0a417277", - "metadata": {}, + "id": "e0bc4c47", + "metadata": { + "editable": true + }, "source": [ "## But noen of these can compete with Newton's method" ] @@ -3756,8 +4319,11 @@ { "cell_type": "code", "execution_count": 30, - "id": "1a541fef", - "metadata": {}, + "id": "43b82587", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Newton's method\n", @@ -3802,8 +4368,10 @@ }, { "cell_type": "markdown", - "id": "9e937e4f", - "metadata": {}, + "id": "6241eaa9", + "metadata": { + "editable": true + }, "source": [ "## Including Stochastic Gradient Descent with Autograd\n", "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**." @@ -3812,8 +4380,11 @@ { "cell_type": "code", "execution_count": 31, - "id": "9afef100", - "metadata": {}, + "id": "f71f3575", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using SGD\n", @@ -3893,8 +4464,10 @@ }, { "cell_type": "markdown", - "id": "2a7e982c", - "metadata": {}, + "id": "8d35483e", + "metadata": { + "editable": true + }, "source": [ "## Same code but now with momentum gradient descent" ] @@ -3902,8 +4475,11 @@ { "cell_type": "code", "execution_count": 32, - "id": "91311a17", - "metadata": {}, + "id": "0759c533", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using SGD\n", @@ -3977,8 +4553,10 @@ }, { "cell_type": "markdown", - "id": "516999d8", - "metadata": {}, + "id": "ccb6f452", + "metadata": { + "editable": true + }, "source": [ "## Similar (second order function now) problem but now with AdaGrad" ] @@ -3986,8 +4564,11 @@ { "cell_type": "code", "execution_count": 33, - "id": "e8292719", - "metadata": {}, + "id": "02dfba31", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n", @@ -4042,16 +4623,20 @@ }, { "cell_type": "markdown", - "id": "bd67f5cb", - "metadata": {}, + "id": "7f7bc3fb", + "metadata": { + "editable": true + }, "source": [ "Running this code we note an almost perfect agreement with the results from matrix inversion." ] }, { "cell_type": "markdown", - "id": "eb0d5fd0", - "metadata": {}, + "id": "afa83d44", + "metadata": { + "editable": true + }, "source": [ "## RMSprop for adaptive learning rate with Stochastic Gradient Descent" ] @@ -4059,8 +4644,11 @@ { "cell_type": "code", "execution_count": 34, - "id": "d2eb93d1", - "metadata": {}, + "id": "5d1d0aff", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n", @@ -4121,8 +4709,10 @@ }, { "cell_type": "markdown", - "id": "669b56c2", - "metadata": {}, + "id": "b7cee753", + "metadata": { + "editable": true + }, "source": [ "## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)" ] @@ -4130,8 +4720,11 @@ { "cell_type": "code", "execution_count": 35, - "id": "bb3f553d", - "metadata": {}, + "id": "b6c78471", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n", @@ -4197,8 +4790,10 @@ }, { "cell_type": "markdown", - "id": "4e5c58ea", - "metadata": {}, + "id": "8f13112c", + "metadata": { + "editable": true + }, "source": [ "## And Logistic Regression" ] @@ -4206,8 +4801,11 @@ { "cell_type": "code", "execution_count": 36, - "id": "1fec659e", - "metadata": {}, + "id": "63dcc21d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4247,8 +4845,10 @@ }, { "cell_type": "markdown", - "id": "ac14943c", - "metadata": {}, + "id": "6bb7b96a", + "metadata": { + "editable": true + }, "source": [ "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n", "\n", @@ -4264,8 +4864,11 @@ { "cell_type": "code", "execution_count": 37, - "id": "57b4e540", - "metadata": {}, + "id": "385bd547", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import jax.numpy as jnp\n", @@ -4280,25 +4883,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.10" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week39/exercisesweek39.do.txt b/doc/src/week39/exercisesweek39.do.txt index 363246a4a..630f7adf0 100644 --- a/doc/src/week39/exercisesweek39.do.txt +++ b/doc/src/week39/exercisesweek39.do.txt @@ -1,13 +1,13 @@ TITLE: Exercises week 39 -AUTHOR: September 25-29, 2023 -DATE: Deadline is Sunday October 1 at midnight +AUTHOR: September 23-27, 2024 +DATE: Deadline is Friday September 27 at midnight ===== Overarching aims of the exercises this week ===== The aim of the exercises this week is to aid you in getting started with writing the report. This will be discussed during the lab -sessions as well. One of the lab sessions will be recorded. +sessions as well. A general guideline can be found at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md". @@ -19,6 +19,6 @@ o Write an abstract for your project o Write an introduction o Include references -Ashort feedback to the this exercise will be available after the deadline. And you can reuse these elements in your final report. +A short feedback to the this exercise will be available before the project deadline. And you can reuse these elements in your final report. diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index e1cd85b94..142aa2ab1 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -6,17 +6,9 @@ DATE: Week 39 ===== Plan for week 39 ===== -!bblock Material for the active learning sessions on Tuesday and Wednesday - * Discussions on how to structure your report for the first project - * Exercise for week 39 on how to write the abstract and the introduction of the report and how to include references. - * Work on project 1, in particular resampling methods like cross-validation and bootstrap. _For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11_. - * "Video on how to write scientific reports recorded during one of the lab sessions":"https://youtu.be/tVW1ZDmZnwM" -These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. - * A general guideline can be found at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md". -!eblock -!bblock Material for the lecture on Thursday September 28 +!bblock Material for the lecture on Monday September 23 * Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent * Stochastic Gradient descent with examples and automatic differentiation * "Video of lecture":"https://youtu.be/bFRVuIJroHs" @@ -29,12 +21,22 @@ These sections summarize neatly what we have done till now and point to what is !eblock +!bblock Material for the active learning sessions on Tuesday and Wednesday + * Discussions on how to structure your report for the first project + * Exercise for week 39 on how to write the abstract and the introduction of the report and how to include references. + * Work on project 1, in particular resampling methods like cross-validation and bootstrap. _For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11_. + * "Video on how to write scientific reports recorded during one of the lab sessions":"https://youtu.be/tVW1ZDmZnwM" +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. + * A general guideline can be found at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md". +!eblock + + # rett opp tyrleif !split -===== Optimization, the central part of any Machine Learning algortithm ===== +===== Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm ===== The first few slides here are a repetition from last week.