diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 65aa91964..248b742e0 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -59,7 +59,6 @@ Automatically generated HTML file from DocOnce source None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), - ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -161,6 +160,7 @@ Automatically generated HTML file from DocOnce source 2, None, 'using-gradient-descent-methods-limitations'), + ('To be added', 2, None, 'to-be-added'), ('Friday October 1', 2, None, 'friday-october-1'), ('Stochastic Gradient Descent', 2, @@ -224,45 +224,45 @@ MathJax.Hub.Config({
  • Optimization, the central part of any Machine Learning algortithm
  • Revisiting our Logistic Regression case
  • The equations to solve
    • -
  • To be added
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • -
  • Convex functions
    • -
  • Convex function
    • -
  • Conditions on convex functions
    • -
  • More on convex functions
    • -
  • Some simple problems
    • -
  • Standard steepest descent
    • -
  • Gradient method
    • -
  • Steepest descent method
    • -
  • Steepest descent method
    • -
  • Final expressions
    • -
  • Steepest descent example
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method and iterations
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Conjugate gradient method
    • -
  • Revisiting our first homework
    • -
  • Gradient descent example
    • -
  • The derivative of the cost/loss function
    • -
  • The Hessian matrix
    • -
  • Simple program
    • -
  • Gradient Descent Example
    • -
  • And a corresponding example using scikit-learn
    • -
  • Gradient descent and Ridge
    • -
  • Program example for gradient descent with Ridge Regression
    • -
  • Using gradient descent methods, limitations
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • +
  • Convex functions
    • +
  • Convex function
    • +
  • Conditions on convex functions
    • +
  • More on convex functions
    • +
  • Some simple problems
    • +
  • Standard steepest descent
    • +
  • Gradient method
    • +
  • Steepest descent method
    • +
  • Steepest descent method
    • +
  • Final expressions
    • +
  • Steepest descent example
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method and iterations
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Conjugate gradient method
    • +
  • Revisiting our first homework
    • +
  • Gradient descent example
    • +
  • The derivative of the cost/loss function
    • +
  • The Hessian matrix
    • +
  • Simple program
    • +
  • Gradient Descent Example
    • +
  • And a corresponding example using scikit-learn
    • +
  • Gradient descent and Ridge
    • +
  • Program example for gradient descent with Ridge Regression
    • +
  • Using gradient descent methods, limitations
    • +
  • To be added
  • Friday October 1
  • Stochastic Gradient Descent
  • Computation of gradients
    • @@ -306,7 +306,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 29, 2021

    +

    Sep 30, 2021


    diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index bedb96af6..a802dee73 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

     
    -

    Sep 29, 2021

    +

    Sep 30, 2021


    @@ -170,6 +170,10 @@ MathJax.Hub.Config({ See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. + +

    +For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. @@ -444,17 +448,6 @@ This defines what is called the Hessian matrix. -

    -

    To be added

    - -

    -We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. - -

    -Make link with linear regression and the Hessian matrix from linear regression. -

    - -

    Solving using Newton-Raphson's method

    @@ -1662,6 +1655,14 @@ plt.show()
    +
    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. +

    + +

    Friday October 1

    diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index 458978331..7d4f92047 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -79,7 +79,6 @@ div { text-align: justify; text-justify: inter-word; } None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), - ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -181,6 +180,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'using-gradient-descent-methods-limitations'), + ('To be added', 2, None, 'to-be-added'), ('Friday October 1', 2, None, 'friday-october-1'), ('Stochastic Gradient Descent', 2, @@ -240,7 +240,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 29, 2021

    +

    Sep 30, 2021












    @@ -256,6 +256,10 @@ MathJax.Hub.Config({ See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +

    +For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. +











    @@ -523,17 +527,6 @@ This defines what is called the Hessian matrix.











    -

    To be added

    - -

    -We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. - -

    -Make link with linear regression and the Hessian matrix from linear regression. - -

    -









    -

    Solving using Newton-Raphson's method

    @@ -1640,6 +1633,14 @@ plt.show()









    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. + +

    +









    +

    Friday October 1

    diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index ed2405a4a..7040e016f 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -84,7 +84,6 @@ div { text-align: justify; text-justify: inter-word; } None, 'revisiting-our-logistic-regression-case'), ('The equations to solve', 2, None, 'the-equations-to-solve'), - ('To be added', 2, None, 'to-be-added'), ("Solving using Newton-Raphson's method", 2, None, @@ -186,6 +185,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'using-gradient-descent-methods-limitations'), + ('To be added', 2, None, 'to-be-added'), ('Friday October 1', 2, None, 'friday-october-1'), ('Stochastic Gradient Descent', 2, @@ -245,7 +245,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 29, 2021

    +

    Sep 30, 2021












    @@ -261,6 +261,10 @@ MathJax.Hub.Config({ See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +

    +For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. +











    @@ -528,17 +532,6 @@ This defines what is called the Hessian matrix.











    -

    To be added

    - -

    -We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. - -

    -Make link with linear regression and the Hessian matrix from linear regression. - -

    -









    -

    Solving using Newton-Raphson's method

    @@ -1645,6 +1638,14 @@ plt.show()









    +

    To be added

    + +

    +We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix. + +

    +









    +

    Friday October 1

    diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index 4f2bb3bb6..428b7dd95 100644 Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 2d9c9b2dd..0da13db9a 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 29, 2021**\n", + "Date: **Sep 30, 2021**\n", "\n", "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -27,6 +27,9 @@ "See [lecture notes for week 39](https://compphysics.github.io/MachineLearning/doc/web/course.html).\n", "For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.\n", "\n", + "**For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11**.\n", + "These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. \n", + "\n", "## Thursday September 30\n", "\n", "[Overview Video, why do we care about gradient methods?](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage)\n", @@ -334,11 +337,6 @@ "This defines what is called the Hessian matrix.\n", "\n", "\n", - "## To be added\n", - "\n", - "We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix.\n", - "\n", - "Make link with linear regression and the Hessian matrix from linear regression.\n", "\n", "\n", "\n", @@ -1891,6 +1889,13 @@ "\n", "* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.\n", "\n", + "## To be added\n", + "\n", + "We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix.\n", + "\n", + "\n", + "\n", + "\n", "## Friday October 1\n", "\n", "\n", diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index 2acc140e7..4cf3be9fa 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -14,6 +14,9 @@ DATE: today See "lecture notes for week 39":"https://compphysics.github.io/MachineLearning/doc/web/course.html". For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +_For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11_. +These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. + !split ===== Thursday September 30 ===== @@ -264,12 +267,6 @@ $p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compa This defines what is called the Hessian matrix. -!split -===== To be added ===== - -We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix. - -Make link with linear regression and the Hessian matrix from linear regression. @@ -1231,6 +1228,14 @@ plt.show() * GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points. +!split +===== To be added ===== + +We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix. + + + + !split ===== Friday October 1 =====