Week 40: From Stochastic Gradient Descent to Neural networks
Contents
Plan for week 40
Overview video on Stochastic Gradient Descent
Batches and mini-batches
Stochastic Gradient Descent (SGD)
Stochastic Gradient Descent
Computation of gradients
SGD example
The gradient step
Simple example code
When do we stop?
Slightly different approach
Program for stochastic gradient
Momentum based GD
More on momentum based approaches
Momentum parameter
Second moment of the gradient
RMS prop
ADAM optimizer
Practical tips
Automatic differentiation
Using autograd
Autograd with more complicated functions
More complicated functions using the elements of their arguments directly
Functions using mathematical functions from Numpy
More autograd
And with loops
Using recursion
Unsupported functions
The syntax a.dot(b) when finding the dot product
Recommended to avoid
Using Autograd with OLS
Including Stochastic Gradient Descent with Autograd
And Logistic Regression
Videos on Neural Networks
Neural networks
Artificial neurons
Neural network types
Feed-forward neural networks
Convolutional Neural Network
Recurrent neural networks
Other types of networks
Multilayer perceptrons
Why multilayer perceptrons?
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
Adding Neural Networks
Mathematical model
Mathematical model
Mathematical model
Mathematical model
Mathematical model
Matrix-vector notation
Matrix-vector notation and activation
Activation functions
Activation functions, Logistic and Hyperbolic ones
Relevance
The multilayer perceptron (MLP)
From one to many layers, the universal approximation theorem
Deriving the back propagation code for a multilayer perceptron model
Definitions
Derivatives and the chain rule
Derivative of the cost function
Bringing it together, first back propagation equation
Derivatives in terms of \( z_j^L \)
Bringing it together
Final back propagating equation
Setting up the Back propagation algorithm
Videos on Neural Networks
Neural Networks demystified
Building Neural Networks from scratch
«
1
...
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
...
69
»