From 58da6c6ba788a3da337834a8eee4bbef51502e6c Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Tue, 25 Oct 2022 15:30:35 +0200 Subject: [PATCH] Update week43.do.txt --- doc/src/week43/week43.do.txt | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/src/week43/week43.do.txt b/doc/src/week43/week43.do.txt index b2d4bf991..f0c8fd270 100644 --- a/doc/src/week43/week43.do.txt +++ b/doc/src/week43/week43.do.txt @@ -30,6 +30,21 @@ DATE: today +neural nets will be very large: impractical to write down gradient formula +by hand for all parameters +● backpropagation = recursive application of the chain rule along a +computational graph to compute the gradients of all +inputs/parameters/intermediates +● implementations maintain a graph structure, where the nodes implement +the forward() / backward() API +● forward: compute result of an operation and save any intermediates +needed for gradient computation in memory +● backward: apply the chain rule to compute the gradient of the loss +function with respect to the inputs + + + + !split ===== CNNs in brief =====