From 43877d18ff4ca2ccb9b7956ba0bc7033cd816111 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 30 Sep 2021 10:31:21 +0200 Subject: [PATCH] update --- doc/pub/week39/html/week39-bs.html | 82 +++++++++---------- doc/pub/week39/html/week39-reveal.html | 25 +++--- doc/pub/week39/html/week39-solarized.html | 27 +++--- doc/pub/week39/html/week39.html | 27 +++--- doc/pub/week39/ipynb/ipynb-week39-src.tar.gz | Bin 193 -> 192 bytes doc/pub/week39/ipynb/week39.ipynb | 17 ++-- doc/src/week39/week39.do.txt | 17 ++-- 7 files changed, 104 insertions(+), 91 deletions(-) diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 65aa91964..248b742e0 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -59,7 +59,6 @@ Automatically generated HTML file from DocOnce source None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), - ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -161,6 +160,7 @@ Automatically generated HTML file from DocOnce source 2, None, 'using-gradient-descent-methods-limitations'), + ('To be added', 2, None, 'to-be-added'), ('Friday October 1', 2, None, 'friday-october-1'), ('Stochastic Gradient Descent', 2, @@ -224,45 +224,45 @@ MathJax.Hub.Config({
  • Optimization, the central part of any Machine Learning algortithm
  • Revisiting our Logistic Regression case
  • The equations to solve
    • -
  • To be added
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Standard steepest descent
    • -
  • Gradient method
    • -
  • Steepest descent method
    • -
  • Steepest descent method
    • -
  • Final expressions
    • -
  • Steepest descent example
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method and iterations
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Gradient descent and Ridge
    • -
  • Program example for gradient descent with Ridge Regression
    • -
  • Using gradient descent methods, limitations
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Standard steepest descent
    • +
  • Gradient method
    • +
  • Steepest descent method
    • +
  • Steepest descent method
    • +
  • Final expressions
    • +
  • Steepest descent example
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method and iterations
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
    • +
  • Gradient descent and Ridge
    • +
  • Program example for gradient descent with Ridge Regression
    • +
  • Using gradient descent methods, limitations
    • +
  • To be added
  • Friday October 1
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -306,7 +306,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 29, 2021

    +

    Sep 30, 2021


    diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index bedb96af6..a802dee73 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

     
    -

    Sep 29, 2021

    +

    Sep 30, 2021


    @@ -170,6 +170,10 @@ MathJax.Hub.Config({ See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. + +

    +For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. @@ -444,17 +448,6 @@ This defines what is called the Hessian matrix. -

    -

    To be added

    - -

    -We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. - -

    -Make link with linear regression and the Hessian matrix from linear regression. -

    - -

    Solving using Newton-Raphson's method

    @@ -1662,6 +1655,14 @@ plt.show()
    +
    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. +

    + +

    Friday October 1

    diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index 458978331..7d4f92047 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -79,7 +79,6 @@ div { text-align: justify; text-justify: inter-word; } None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), - ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -181,6 +180,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'using-gradient-descent-methods-limitations'), + ('To be added', 2, None, 'to-be-added'), ('Friday October 1', 2, None, 'friday-october-1'), ('Stochastic Gradient Descent', 2, @@ -240,7 +240,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 29, 2021

    +

    Sep 30, 2021












    @@ -256,6 +256,10 @@ MathJax.Hub.Config({ See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +

    +For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. +











    @@ -523,17 +527,6 @@ This defines what is called the Hessian matrix.











    -

    To be added

    - -

    -We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. - -

    -Make link with linear regression and the Hessian matrix from linear regression. - -

    -









    -

    Solving using Newton-Raphson's method

    @@ -1640,6 +1633,14 @@ plt.show()









    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. + +

    +









    +

    Friday October 1

    diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index ed2405a4a..7040e016f 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -84,7 +84,6 @@ div { text-align: justify; text-justify: inter-word; } None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), - ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -186,6 +185,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'using-gradient-descent-methods-limitations'), + ('To be added', 2, None, 'to-be-added'), ('Friday October 1', 2, None, 'friday-october-1'), ('Stochastic Gradient Descent', 2, @@ -245,7 +245,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 29, 2021

    +

    Sep 30, 2021












    @@ -261,6 +261,10 @@ MathJax.Hub.Config({ See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +

    +For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. +











    @@ -528,17 +532,6 @@ This defines what is called the Hessian matrix.











    -

    To be added

    - -

    -We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. - -

    -Make link with linear regression and the Hessian matrix from linear regression. - -

    -









    -

    Solving using Newton-Raphson's method

    @@ -1645,6 +1638,14 @@ plt.show()









    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. + +

    +









    +

    Friday October 1

    diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index 4f2bb3bb6f06cdecc621fff094d5f07d6943d602..428b7dd95c2c23c9ccdc43ab20084f957a0d632d 100644 GIT binary patch literal 192 zcmV;x06+g9iwFS9byZ;i1MSaC3c@fD2H>uHia9|^(ibiTyKo_hc!88oZLCddlA^u6 zeSoeMH${Yeo1bBZVdjvnH@hsbck3;N5Rya@jF~3!l!VRogwhyjL|M#ImQn_UhA~S# zAj_Tf(izM3a7t61Q5ux*=7zDd{IF+!1)ljQj+HR5-S@Uq5~Q-2D^vLv1MSbv3W7io2XN0m#XN!Rs;hJi^3WlO=ml0Bb2E2sccpy$ z{D3+YT_l42UH%L+3^Rvpz1d}fy<2ZFgpeeTVazmPQ<8A5CzLXvF%20BQ^FD;G-4t1 zfGl^?OJ^+C!zoR5MrlyKn;XW;^246_6?o>KI99^IcHi4dNs!87u2c;-#5!69qU~i4 vg+ep7K;yL&8iC6mcu@!|l;jt`)#{{qV*>xz&p3|bIA41Jc`nMR00;m8rU+Fg diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 2d9c9b2dd..0da13db9a 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 29, 2021**\n", + "Date: **Sep 30, 2021**\n", "\n", "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -27,6 +27,9 @@ "See [lecture notes for week 39](https://compphysics.github.io/MachineLearning/doc/web/course.html).\n", "For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.\n", "\n", + "**For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11**.\n", + "These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. \n", + "\n", "## Thursday September 30\n", "\n", "[Overview Video, why do we care about gradient methods?](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage)\n", @@ -334,11 +337,6 @@ "This defines what is called the Hessian matrix.\n", "\n", "\n", - "## To be added\n", - "\n", - "We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix.\n", - "\n", - "Make link with linear regression and the Hessian matrix from linear regression.\n", "\n", "\n", "\n", @@ -1891,6 +1889,13 @@ "\n", "* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.\n", "\n", + "## To be added\n", + "\n", + "We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix.\n", + "\n", + "\n", + "\n", + "\n", "## Friday October 1\n", "\n", "\n", diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index 2acc140e7..4cf3be9fa 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -14,6 +14,9 @@ DATE: today See "lecture notes for week 39":"https://compphysics.github.io/MachineLearning/doc/web/course.html". For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +_For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11_. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. + !split ===== Thursday September 30 ===== @@ -264,12 +267,6 @@ $p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compa This defines what is called the Hessian matrix. -!split -===== To be added ===== - -We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix. - -Make link with linear regression and the Hessian matrix from linear regression. @@ -1231,6 +1228,14 @@ plt.show() * GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points. +!split +===== To be added ===== + +We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix. + + + + !split ===== Friday October 1 =====