diff --git a/doc/pub/week42/html/._week42-bs000.html b/doc/pub/week42/html/._week42-bs000.html index 01e8a9642..d478d6ac7 100644 --- a/doc/pub/week42/html/._week42-bs000.html +++ b/doc/pub/week42/html/._week42-bs000.html @@ -36,16 +36,17 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
  • Lecture October 14, 2024
    • -
  • Material for the active learning sessions on Tuesday and Wednesday
    • -
  • Writing a code which implements a feed-forward neural network
    • -
  • Mathematics of deep learning
    • -
  • Reminder on books with hands-on material and codes
    • -
  • Reading recommendations
    • -
  • First network example, simple percepetron with one input
    • -
  • Layout of a simple neural network with no hidden layer
    • -
  • Optimizing the parameters
    • -
  • Adding a hidden layer
    • -
  • Layout of a simple neural network with one hidden layer
    • -
  • The derivatives
    • -
  • Important observations
    • -
  • The training
    • -
  • Code example
    • -
  • Simple neural network and the back propagation equations
    • -
  • Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node
    • -
  • The ouput layer
    • -
  • Compact expressions
    • -
  • Output layer
    • -
  • Explicit derivatives
    • -
  • Derivatives of the hidden layer
    • -
  • Final expression
    • -
  • Completing the list
    • -
  • Final expressions for the biases of the hidden layer
    • -
  • Gradient expressions
    • -
  • Setting up the equations for a neural network
    • -
  • Layout of a neural network with three hidden layers (last later = \( l=L=4 \), first layer \( l=0 \))
    • -
  • Definitions
    • -
  • Inputs to the activation function
    • -
  • Layout of input to first hidden layer \( l=1 \) from input layer \( l=0 \)
    • -
  • Derivatives and the chain rule
    • -
  • Derivative of the cost function
    • -
  • The back propagation equations for a neural network
    • -
  • Analyzing the last results
    • -
  • More considerations
    • -
  • Derivatives in terms of \( z_j^L \)
    • -
  • Bringing it together
    • -
  • Final back propagating equation
    • -
  • Using the chain rule and summing over all \( k \) entries
    • -
  • Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations
    • -
  • Setting up the back propagation algorithm, part 1
    • -
  • Setting up the back propagation algorithm, part 2
    • -
  • Setting up the Back propagation algorithm, part 3
    • -
  • Updating the gradients
    • +
  • Lecture October 13, 2025
    • +
  • Readings and videos
    • +
  • Material for the lab sessions on Tuesday and Wednesday
    • +
  • Lecture material: Writing a code which implements a feed-forward neural network
    • +
  • Mathematics of deep learning
    • +
  • Reminder on books with hands-on material and codes
    • +
  • Reading recommendations
    • +
  • Reminder from last week: First network example, simple percepetron with one input
    • +
  • Layout of a simple neural network with no hidden layer
    • +
  • Optimizing the parameters
    • +
  • Adding a hidden layer
    • +
  • Layout of a simple neural network with one hidden layer
    • +
  • The derivatives
    • +
  • Important observations
    • +
  • The training
    • +
  • Code example
    • +
  • Simple neural network and the back propagation equations
    • +
  • Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node
    • +
  • The ouput layer
    • +
  • Compact expressions
    • +
  • Output layer
    • +
  • Explicit derivatives
    • +
  • Derivatives of the hidden layer
    • +
  • Final expression
    • +
  • Completing the list
    • +
  • Final expressions for the biases of the hidden layer
    • +
  • Gradient expressions
    • +
  • Setting up the equations for a neural network
    • +
  • Layout of a neural network with three hidden layers (last layer = \( l=L=4 \), first layer \( l=0 \))
    • +
  • Definitions
    • +
  • Inputs to the activation function
    • +
  • Layout of input to first hidden layer \( l=1 \) from input layer \( l=0 \)
    • +
  • Derivatives and the chain rule
    • +
  • Derivative of the cost function
    • +
  • The back propagation equations for a neural network
    • +
  • Analyzing the last results
    • +
  • More considerations
    • +
  • Derivatives in terms of \( z_j^L \)
    • +
  • Bringing it together
    • +
  • Final back propagating equation
    • +
  • Using the chain rule and summing over all \( k \) entries
    • +
  • Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations
    • +
  • Setting up the back propagation algorithm, part 1
    • +
  • Setting up the back propagation algorithm, part 2
    • +
  • Setting up the Back propagation algorithm, part 3
    • +
  • Updating the gradients
  • Activation functions
    • -
  •    Activation functions, Logistic and Hyperbolic ones
    • -
  • Relevance
    • -
  • Vanishing gradients
    • -
  • Exploding gradients
    • -
  • Is the Logistic activation function (Sigmoid) our choice?
    • -
  • Logistic function as the root of problems
    • -
  • The derivative of the Logistic funtion
    • -
  • Insights from the paper by Glorot and Bengio
    • -
  • The RELU function family
    • -
  • ELU function
    • -
  • Which activation function should we use?
    • -
  • More on activation functions, output layers
    • -
  • Fine-tuning neural network hyperparameters
    • -
  • Hidden layers
    • -
  • Batch Normalization
    • -
  • Dropout
    • -
  • Gradient Clipping
    • -
  • A top-down perspective on Neural networks
    • -
  • More top-down perspectives
    • -
  • Limitations of supervised learning with deep networks
    • -
  • Limitations of NNs
    • -
  • Homogeneous data
    • -
  • More limitations
    • -
  • Setting up a Multi-layer perceptron model for classification
    • -
  • Defining the cost function
    • -
  • Example: binary classification problem
    • -
  • The Softmax function
    • -
  • Developing a code for doing neural networks with back propagation
    • -
  • Collect and pre-process data
    • -
  • Train and test datasets
    • -
  • Define model and architecture
    • -
  • Layers
    • -
  • Weights and biases
    • -
  • Feed-forward pass
    • -
  • Matrix multiplications
    • -
  • Choose cost function and optimizer
    • -
  • Optimizing the cost function
    • -
  • Regularization
    • -
  • Matrix multiplication
    • -
  • Improving performance
    • -
  • Full object-oriented implementation
    • -
  • Evaluate model performance on test data
    • -
  • Adjust hyperparameters
    • -
  • Visualization
    • -
  • scikit-learn implementation
    • -
  • Visualization
    • -
  • Building neural networks in Tensorflow and Keras
    • -
  • Tensorflow
    • -
  • Using Keras
    • -
  • Collect and pre-process data
    • -
  • The Breast Cancer Data, now with Keras
    • +
  •    Activation functions, Logistic and Hyperbolic ones
    • +
  • Relevance
    • +
  • Vanishing gradients
    • +
  • Exploding gradients
    • +
  • Is the Logistic activation function (Sigmoid) our choice?
    • +
  • Logistic function as the root of problems
    • +
  • The derivative of the Logistic funtion
    • +
  • Insights from the paper by Glorot and Bengio
    • +
  • The RELU function family
    • +
  • ELU function
    • +
  • Which activation function should we use?
    • +
  • More on activation functions, output layers
    • +
  • Fine-tuning neural network hyperparameters
    • +
  • Hidden layers
    • +
  • Batch Normalization
    • +
  • Dropout
    • +
  • Gradient Clipping
    • +
  • A top-down perspective on Neural networks
    • +
  • More top-down perspectives
    • +
  • Limitations of supervised learning with deep networks
    • +
  • Limitations of NNs
    • +
  • Homogeneous data
    • +
  • More limitations
    • +
  • Setting up a Multi-layer perceptron model for classification
    • +
  • Defining the cost function
    • +
  • Example: binary classification problem
    • +
  • The Softmax function
    • +
  • Developing a code for doing neural networks with back propagation
    • +
  • Collect and pre-process data
    • +
  • Train and test datasets
    • +
  • Define model and architecture
    • +
  • Layers
    • +
  • Weights and biases
    • +
  • Feed-forward pass
    • +
  • Matrix multiplications
    • +
  • Choose cost function and optimizer
    • +
  • Optimizing the cost function
    • +
  • Regularization
    • +
  • Matrix multiplication
    • +
  • Improving performance
    • +
  • Full object-oriented implementation
    • +
  • Evaluate model performance on test data
    • +
  • Adjust hyperparameters
    • +
  • Visualization
    • +
  • scikit-learn implementation
    • +
  • Visualization
    • +
  • Building neural networks in Tensorflow and Keras
    • +
  • Tensorflow
    • +
  • Using Keras
    • +
  • Collect and pre-process data
  • Building a neural network code
  •    Learning rate methods
  •    Usage of the above learning rate schedulers
    • @@ -472,18 +470,15 @@ MathJax.Hub.Config({
    -Morten Hjorth-Jensen [1, 2] +Morten Hjorth-Jensen
    - +
    -[1] Department of Physics, University of Oslo -
    -
    -[2] Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University +Department of Physics, University of Oslo, Norway

    -

    October 14-18, 2024

    +

    October 13-17, 2025


    @@ -522,7 +517,7 @@ MathJax.Hub.Config({ -->
    - © 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2025, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    diff --git a/doc/pub/week42/html/._week42-bs001.html b/doc/pub/week42/html/._week42-bs001.html index 03cafa1e4..31d85ff9e 100644 --- a/doc/pub/week42/html/._week42-bs001.html +++ b/doc/pub/week42/html/._week42-bs001.html @@ -36,16 +36,17 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
  • Lecture October 14, 2024
    • -
  • Material for the active learning sessions on Tuesday and Wednesday
    • -
  • Writing a code which implements a feed-forward neural network
    • -
  • Mathematics of deep learning
    • -
  • Reminder on books with hands-on material and codes
    • -
  • Reading recommendations
    • -
  • First network example, simple percepetron with one input
    • -
  • Layout of a simple neural network with no hidden layer
    • -
  • Optimizing the parameters
    • -
  • Adding a hidden layer
    • -
  • Layout of a simple neural network with one hidden layer
    • -
  • The derivatives
    • -
  • Important observations
    • -
  • The training
    • -
  • Code example
    • -
  • Simple neural network and the back propagation equations
    • -
  • Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node
    • -
  • The ouput layer
    • -
  • Compact expressions
    • -
  • Output layer
    • -
  • Explicit derivatives
    • -
  • Derivatives of the hidden layer
    • -
  • Final expression
    • -
  • Completing the list
    • -
  • Final expressions for the biases of the hidden layer
    • -
  • Gradient expressions
    • -
  • Setting up the equations for a neural network
    • -
  • Layout of a neural network with three hidden layers (last later = \( l=L=4 \), first layer \( l=0 \))
    • -
  • Definitions
    • -
  • Inputs to the activation function
    • -
  • Layout of input to first hidden layer \( l=1 \) from input layer \( l=0 \)
    • -
  • Derivatives and the chain rule
    • -
  • Derivative of the cost function
    • -
  • The back propagation equations for a neural network
    • -
  • Analyzing the last results
    • -
  • More considerations
    • -
  • Derivatives in terms of \( z_j^L \)
    • -
  • Bringing it together
    • -
  • Final back propagating equation
    • -
  • Using the chain rule and summing over all \( k \) entries
    • -
  • Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations
    • -
  • Setting up the back propagation algorithm, part 1
    • -
  • Setting up the back propagation algorithm, part 2
    • -
  • Setting up the Back propagation algorithm, part 3
    • -
  • Updating the gradients
    • +
  • Lecture October 13, 2025
    • +
  • Readings and videos
    • +
  • Material for the lab sessions on Tuesday and Wednesday
    • +
  • Lecture material: Writing a code which implements a feed-forward neural network
    • +
  • Mathematics of deep learning
    • +
  • Reminder on books with hands-on material and codes
    • +
  • Reading recommendations
    • +
  • Reminder from last week: First network example, simple percepetron with one input
    • +
  • Layout of a simple neural network with no hidden layer
    • +
  • Optimizing the parameters
    • +
  • Adding a hidden layer
    • +
  • Layout of a simple neural network with one hidden layer
    • +
  • The derivatives
    • +
  • Important observations
    • +
  • The training
    • +
  • Code example
    • +
  • Simple neural network and the back propagation equations
    • +
  • Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node
    • +
  • The ouput layer
    • +
  • Compact expressions
    • +
  • Output layer
    • +
  • Explicit derivatives
    • +
  • Derivatives of the hidden layer
    • +
  • Final expression
    • +
  • Completing the list
    • +
  • Final expressions for the biases of the hidden layer
    • +
  • Gradient expressions
    • +
  • Setting up the equations for a neural network
    • +
  • Layout of a neural network with three hidden layers (last layer = \( l=L=4 \), first layer \( l=0 \))
    • +
  • Definitions
    • +
  • Inputs to the activation function
    • +
  • Layout of input to first hidden layer \( l=1 \) from input layer \( l=0 \)
    • +
  • Derivatives and the chain rule
    • +
  • Derivative of the cost function
    • +
  • The back propagation equations for a neural network
    • +
  • Analyzing the last results
    • +
  • More considerations
    • +
  • Derivatives in terms of \( z_j^L \)
    • +
  • Bringing it together
    • +
  • Final back propagating equation
    • +
  • Using the chain rule and summing over all \( k \) entries
    • +
  • Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations
    • +
  • Setting up the back propagation algorithm, part 1
    • +
  • Setting up the back propagation algorithm, part 2
    • +
  • Setting up the Back propagation algorithm, part 3
    • +
  • Updating the gradients
  • Activation functions
    • -
  •    Activation functions, Logistic and Hyperbolic ones
    • -
  • Relevance
    • -
  • Vanishing gradients
    • -
  • Exploding gradients
    • -
  • Is the Logistic activation function (Sigmoid) our choice?
    • -
  • Logistic function as the root of problems
    • -
  • The derivative of the Logistic funtion
    • -
  • Insights from the paper by Glorot and Bengio
    • -
  • The RELU function family
    • -
  • ELU function
    • -
  • Which activation function should we use?
    • -
  • More on activation functions, output layers
    • -
  • Fine-tuning neural network hyperparameters
    • -
  • Hidden layers
    • -
  • Batch Normalization
    • -
  • Dropout
    • -
  • Gradient Clipping
    • -
  • A top-down perspective on Neural networks
    • -
  • More top-down perspectives
    • -
  • Limitations of supervised learning with deep networks
    • -
  • Limitations of NNs
    • -
  • Homogeneous data
    • -
  • More limitations
    • -
  • Setting up a Multi-layer perceptron model for classification
    • -
  • Defining the cost function
    • -
  • Example: binary classification problem
    • -
  • The Softmax function
    • -
  • Developing a code for doing neural networks with back propagation
    • -
  • Collect and pre-process data
    • -
  • Train and test datasets
    • -
  • Define model and architecture
    • -
  • Layers
    • -
  • Weights and biases
    • -
  • Feed-forward pass
    • -
  • Matrix multiplications
    • -
  • Choose cost function and optimizer
    • -
  • Optimizing the cost function
    • -
  • Regularization
    • -
  • Matrix multiplication
    • -
  • Improving performance
    • -
  • Full object-oriented implementation
    • -
  • Evaluate model performance on test data
    • -
  • Adjust hyperparameters
    • -
  • Visualization
    • -
  • scikit-learn implementation
    • -
  • Visualization
    • -
  • Building neural networks in Tensorflow and Keras
    • -
  • Tensorflow
    • -
  • Using Keras
    • -
  • Collect and pre-process data
    • -
  • The Breast Cancer Data, now with Keras
    • +
  •    Activation functions, Logistic and Hyperbolic ones
    • +
  • Relevance
    • +
  • Vanishing gradients
    • +
  • Exploding gradients
    • +
  • Is the Logistic activation function (Sigmoid) our choice?
    • +
  • Logistic function as the root of problems
    • +
  • The derivative of the Logistic funtion
    • +
  • Insights from the paper by Glorot and Bengio
    • +
  • The RELU function family
    • +
  • ELU function
    • +
  • Which activation function should we use?
    • +
  • More on activation functions, output layers
    • +
  • Fine-tuning neural network hyperparameters
    • +
  • Hidden layers
    • +
  • Batch Normalization
    • +
  • Dropout
    • +
  • Gradient Clipping
    • +
  • A top-down perspective on Neural networks
    • +
  • More top-down perspectives
    • +
  • Limitations of supervised learning with deep networks
    • +
  • Limitations of NNs
    • +
  • Homogeneous data
    • +
  • More limitations
    • +
  • Setting up a Multi-layer perceptron model for classification
    • +
  • Defining the cost function
    • +
  • Example: binary classification problem
    • +
  • The Softmax function
    • +
  • Developing a code for doing neural networks with back propagation
    • +
  • Collect and pre-process data
    • +
  • Train and test datasets
    • +
  • Define model and architecture
    • +
  • Layers
    • +
  • Weights and biases
    • +
  • Feed-forward pass
    • +
  • Matrix multiplications
    • +
  • Choose cost function and optimizer
    • +
  • Optimizing the cost function
    • +
  • Regularization
    • +
  • Matrix multiplication
    • +
  • Improving performance
    • +
  • Full object-oriented implementation
    • +
  • Evaluate model performance on test data
    • +
  • Adjust hyperparameters
    • +
  • Visualization
    • +
  • scikit-learn implementation
    • +
  • Visualization
    • +
  • Building neural networks in Tensorflow and Keras
    • +
  • Tensorflow
    • +
  • Using Keras
    • +
  • Collect and pre-process data
  • Building a neural network code
  •    Learning rate methods
  •    Usage of the above learning rate schedulers
    • @@ -465,34 +463,17 @@ MathJax.Hub.Config({

     

     

     

    -

    Lecture October 14, 2024

    +

    Lecture October 13, 2025

    1. Building our own Feed-forward Neural Network and discussion of project 2
      2. +
    2. Project 2 is available at https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/2025/Project2/ipynb/Project2.ipynb
    -
    -
    - -
      -
    1. These lecture notes
      2. -
    2. Video of lecture
      3. -
    3. Whiteboard notes
      4. -
    4. For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7.
      5. -
    5. Neural Networks demystified at https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs
      6. -
    6. Building Neural Networks from scratch at https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex
      7. -
    7. Video on Neural Networks at https://www.youtube.com/watch?v=CqOfi41LfDw
      8. -
    8. Video on the back propagation algorithm at https://www.youtube.com/watch?v=Ilg3gGewQ5U
      9. -
    -

    I also recommend Michael Nielsen's intuitive approach to the neural networks and the universal approximation theorem, see the slides at http://neuralnetworksanddeeplearning.com/chap4.html.

    -
    -
    - -