lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 40

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-09-29 21:22:43 +02:00
parent 9c19ea9bfd
commit efdc7bd11e
73 changed files with 7517 additions and 6477 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week40/week40.do.txt
+26 -21
View File
@@ -1,6 +1,6 @@
TITLE: Week 40: Gradient descent methods (continued) and start Neural networks
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo, Norway & Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, USA
DATE: October 2-6, 2023
DATE: September 30-October 4, 2024
@@ -8,32 +8,37 @@ DATE: October 2-6, 2023
===== Plans for week 40 =====
!split
===== Lecture Monday September 30, 2024 =====
!bblock
o Stochastic Gradient descent with examples and automatic differentiation
o If we get time, we start with the basics of Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model
# * "Video of lecture":"https://youtu.be/75pr3hKY20U"
# * "Whiteboard notes at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesOct5.pdf"
!eblock
!split
===== Suggested readings and videos =====
!bblock Readings and Videos:
o The lecture notes for week 40 (these notes)
o For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.
o For neural networks we recommend Goodfellow et al chapter 6 and Raschka et al pages 48-60
o Video on gradient descent at URL:"https://www.youtube.com/watch?v=sDv4f4s2SB8"
o Video on stochastic gradient descent at URL:"https://www.youtube.com/watch?v=vMh0zPT0tLI"
o Neural Networks demystified at URL:"https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs"
o Building Neural Networks from scratch at URL:https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex"
!eblock
!split
===== Lab sessions Tuesday and Wednesday =====
!bblock Material for the active learning sessions on Tuesday and Wednesday
* Work on project 1 and discussions on how to structure your report
* No weekly exercises for week 40, project work only
* "Video on how to write scientific reports recorded during one of the lab sessions":"https://youtu.be/tVW1ZDmZnwM"
* Video on how to write scientific reports recorded during one of the lab sessions at URL:"https://youtu.be/tVW1ZDmZnwM"
* A general guideline can be found at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md".
!eblock
!bblock Material for the lecture on Thursday October 5, 2023
* Stochastic Gradient descent with examples and automatic differentiation
* Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.
* "Video of lecture":"https://youtu.be/75pr3hKY20U"
* "Whiteboard notes at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesOct5.pdf"
* Readings and Videos:
* These lecture notes
* For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.
* "Aurelien Geron's chapter 4 on stochastic gradient descent":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf"
* For neural networks we recommend Goodfellow et al chapter 6.
* "Video on gradient descent":"https://www.youtube.com/watch?v=sDv4f4s2SB8"
* "Video on stochastic gradient descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI"
* "Neural Networks demystified":"https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs"
* "Building Neural Networks from scratch":"https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex"
!eblock
!split
===== Summary from last week, using gradient descent methods, limitations =====
@@ -489,7 +494,7 @@ adaptively change the step size to match the landscape without paying
the steep computational price of calculating or approximating
Hessians.
Recently, a number of methods have been introduced that accomplish
During the last decade a number of methods have been introduced that accomplish
this by tracking not only the gradient, but also the second moment of
the gradient. These methods include AdaGrad, AdaDelta, Root Mean Squared Propagation (RMS-Prop), and
"ADAM":"https://arxiv.org/abs/1412.6980".