diff --git a/doc/src/week43/week43.do.txt b/doc/src/week43/week43.do.txt index b2d4bf991..f0c8fd270 100644 --- a/doc/src/week43/week43.do.txt +++ b/doc/src/week43/week43.do.txt @@ -30,6 +30,21 @@ DATE: today +neural nets will be very large: impractical to write down gradient formula +by hand for all parameters +● backpropagation = recursive application of the chain rule along a +computational graph to compute the gradients of all +inputs/parameters/intermediates +● implementations maintain a graph structure, where the nodes implement +the forward() / backward() API +● forward: compute result of an operation and save any intermediates +needed for gradient computation in memory +● backward: apply the chain rule to compute the gradient of the loss +function with respect to the inputs + + + + !split ===== CNNs in brief =====