From 05c1a5b9794c617a3381f91fabbd14d67e2ca8fc Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 14 Oct 2021 11:21:46 +0200 Subject: [PATCH] added files --- doc/pub/week40/html/._week40-bs061.html | 413 ++++++++++++++++++++++ doc/pub/week40/html/._week40-bs062.html | 379 +++++++++++++++++++++ doc/pub/week40/html/._week40-bs063.html | 423 +++++++++++++++++++++++ doc/pub/week40/html/._week40-bs064.html | 392 +++++++++++++++++++++ doc/pub/week40/html/._week40-bs065.html | 435 ++++++++++++++++++++++++ 5 files changed, 2042 insertions(+) create mode 100644 doc/pub/week40/html/._week40-bs061.html create mode 100644 doc/pub/week40/html/._week40-bs062.html create mode 100644 doc/pub/week40/html/._week40-bs063.html create mode 100644 doc/pub/week40/html/._week40-bs064.html create mode 100644 doc/pub/week40/html/._week40-bs065.html diff --git a/doc/pub/week40/html/._week40-bs061.html b/doc/pub/week40/html/._week40-bs061.html new file mode 100644 index 000000000..17f9e3d14 --- /dev/null +++ b/doc/pub/week40/html/._week40-bs061.html @@ -0,0 +1,413 @@ + + + + + + + + +Week 40: From Stochastic Gradient Descent to Neural networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Week 40: From Stochastic Gradient Descent to Neural networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Bringing it together, first back propagation equation

+ +

+We have thus +$$ +\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \left(a_j^L - t_j\right)a_j^L(1-a_j^L)a_k^{L-1}, +$$ + +

+Defining +$$ +\delta_j^L = a_j^L(1-a_j^L)\left(a_j^L - t_j\right) = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}, +$$ + +and using the Hadamard product of two vectors we can write this as +$$ +\hat{\delta}^L = f'(\hat{z}^L)\circ\frac{\partial {\cal C}}{\partial (\hat{a}^L)}. +$$ + +

+This is an important expression. The second term on the right handside +measures how fast the cost function is changing as a function of the $j$th +output activation. If, for example, the cost function doesn't depend +much on a particular output node \( j \), then \( \delta_j^L \) will be small, +which is what we would expect. The first term on the right, measures +how fast the activation function \( f \) is changing at a given activation +value \( z_j^L \). + +

+Notice that everything in the above equations is easily computed. In +particular, we compute \( z_j^L \) while computing the behaviour of the +network, and it is only a small additional overhead to compute +\( f'(z^L_j) \). The exact form of the derivative with respect to the +output depends on the form of the cost function. +However, provided the cost function is known there should be little +trouble in calculating + +$$ +\frac{\partial {\cal C}}{\partial (a_j^L)} +$$ + +

+With the definition of \( \delta_j^L \) we have a more compact definition of the derivative of the cost function in terms of the weights, namely +$$ +\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \delta_j^La_k^{L-1}. +$$ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/week40/html/._week40-bs062.html b/doc/pub/week40/html/._week40-bs062.html new file mode 100644 index 000000000..4d717fafe --- /dev/null +++ b/doc/pub/week40/html/._week40-bs062.html @@ -0,0 +1,379 @@ + + + + + + + + +Week 40: From Stochastic Gradient Descent to Neural networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Week 40: From Stochastic Gradient Descent to Neural networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Derivatives in terms of \( z_j^L \)

+ +

+It is also easy to see that our previous equation can be written as + +$$ +\delta_j^L =\frac{\partial {\cal C}}{\partial z_j^L}= \frac{\partial {\cal C}}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L}, +$$ + +which can also be interpreted as the partial derivative of the cost function with respect to the biases \( b_j^L \), namely +$$ +\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}\frac{\partial b_j^L}{\partial z_j^L}=\frac{\partial {\cal C}}{\partial b_j^L}, +$$ + +That is, the error \( \delta_j^L \) is exactly equal to the rate of change of the cost function as a function of the bias. +

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/week40/html/._week40-bs063.html b/doc/pub/week40/html/._week40-bs063.html new file mode 100644 index 000000000..a5d98fa4a --- /dev/null +++ b/doc/pub/week40/html/._week40-bs063.html @@ -0,0 +1,423 @@ + + + + + + + + +Week 40: From Stochastic Gradient Descent to Neural networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Week 40: From Stochastic Gradient Descent to Neural networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Bringing it together

+ +

+We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are + +

+

+
+

+ +$$ +\begin{equation} +\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \delta_j^La_k^{L-1}, +\tag{18} +\end{equation} +$$ + +and +$$ +\begin{equation} +\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}, +\tag{19} +\end{equation} +$$ + +and + +$$ +\begin{equation} +\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}, +\tag{20} +\end{equation} +$$ +

+
+ + +

+An interesting consequence of the above equations is that when the +activation \( a_k^{L-1} \) is small, the gradient term, that is the +derivative of the cost function with respect to the weights, will also +tend to be small. We say then that the weight learns slowly, meaning +that it changes slowly when we minimize the weights via say gradient +descent. In this case we say the system learns slowly. + +

+Another interesting feature is that is when the activation function, +represented by the sigmoid function here, is rather flat when we move towards +its end values \( 0 \) and \( 1 \) (see the above Python codes). In these +cases, the derivatives of the activation function will also be close +to zero, meaning again that the gradients will be small and the +network learns slowly again. + +

+We need a fourth equation and we are set. We are going to propagate +backwards in order to the determine the weights and biases. In order +to do so we need to represent the error in the layer before the final +one \( L-1 \) in terms of the errors in the final output layer. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/week40/html/._week40-bs064.html b/doc/pub/week40/html/._week40-bs064.html new file mode 100644 index 000000000..ce86adb50 --- /dev/null +++ b/doc/pub/week40/html/._week40-bs064.html @@ -0,0 +1,392 @@ + + + + + + + + +Week 40: From Stochastic Gradient Descent to Neural networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Week 40: From Stochastic Gradient Descent to Neural networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Final back propagating equation

+ +

+We have that (replacing \( L \) with a general layer \( l \)) +$$ +\delta_j^l =\frac{\partial {\cal C}}{\partial z_j^l}. +$$ + +We want to express this in terms of the equations for layer \( l+1 \). Using the chain rule and summing over all \( k \) entries we have + +$$ +\delta_j^l =\sum_k \frac{\partial {\cal C}}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}=\sum_k \delta_k^{l+1}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}, +$$ + +and recalling that +$$ +z_j^{l+1} = \sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1}, +$$ + +with \( M_l \) being the number of nodes in layer \( l \), we obtain +$$ +\delta_j^l =\sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l), +$$ + +This is our final equation. + +

+We are now ready to set up the algorithm for back propagation and learning the weights and biases. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/week40/html/._week40-bs065.html b/doc/pub/week40/html/._week40-bs065.html new file mode 100644 index 000000000..26d83458d --- /dev/null +++ b/doc/pub/week40/html/._week40-bs065.html @@ -0,0 +1,435 @@ + + + + + + + + +Week 40: From Stochastic Gradient Descent to Neural networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Week 40: From Stochastic Gradient Descent to Neural networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Setting up the Back propagation algorithm

+ +

+The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm. + +

+

+
+

+First, we set up the input data \( \hat{x} \) and the activations +\( \hat{z}_1 \) of the input layer and compute the activation function and +the pertinent outputs \( \hat{a}^1 \). +

+
+ + +

+

+
+

+Secondly, we perform then the feed forward till we reach the output +layer and compute all \( \hat{z}_l \) of the input layer and compute the +activation function and the pertinent outputs \( \hat{a}^l \) for +\( l=2,3,\dots,L \). +

+
+ + +

+

+
+

+Thereafter we compute the ouput error \( \hat{\delta}^L \) by computing all +$$ +\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}. +$$ +

+
+ + +

+

+
+

+Then we compute the back propagate error for each \( l=L-1,L-2,\dots,2 \) as +$$ +\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l). +$$ +

+
+ + +

+

+
+

+Finally, we update the weights and the biases using gradient descent for each \( l=L-1,L-2,\dots,2 \) and update the weights and biases according to the rules +$$ +w_{jk}^l\leftarrow = w_{jk}^l- \eta \delta_j^la_k^{l-1}, +$$ + + +$$ +b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l, +$$ +

+
+ + +

+The parameter \( \eta \) is the learning parameter discussed in connection with the gradient descent methods. +Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training. + +

+ +

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + +