update

doc/src/week42/week42.do.txt

@@ -26,6 +26,7 @@ I also  recommend Michael Nielsen's intuitive approach to the neural networks an
   * Discussion of project 2
 !eblock  
 
+_Note_: some of the codes will also be discussed next week in connection with the solution of differential equations.
 
 
 !split
@@ -34,7 +35,7 @@ I also  recommend Michael Nielsen's intuitive approach to the neural networks an
 Last week we discussed the basics of neural networks and deep learning
 and the basics of automatic differentiation.  We looked also at
 examples on how compute the parameters of a simple network with scalar
-inputs and ouputs and no or just one hiden layers.
+inputs and ouputs and no or just one hidden layers.
 
 
 We ended our discussions with the derivation of the equations for a
@@ -576,7 +577,7 @@ network $\bm{\tilde{y}}$ and the inputs $\bm{x}$ we
 define now the activation $z_j^l$ of node/neuron/unit $j$ of the
 $l$-th layer as a function of the bias, the weights which add up from
 the previous layer $l-1$ and the forward passes/outputs
-$\hat{a}^{l-1}$ from the previous layer as
+$\bm{a}^{l-1}$ from the previous layer as
 
 
 !bt
@@ -684,7 +685,7 @@ Defining
 and using the Hadamard product of two vectors we can write this as
 !bt
 \[
-\bm{\delta}^L = \sigma'(\hat{z}^L)\circ\frac{\partial {\cal C}}{\partial (\bm{a}^L)}.
+\bm{\delta}^L = \sigma'(\bm{z}^L)\circ\frac{\partial {\cal C}}{\partial (\bm{a}^L)}.
 \]
 !et
 
@@ -720,7 +721,7 @@ trouble in calculating
 With the definition of $\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely
 !bt
 \[
-\frac{\partial{\cal C}}{\partial w_{jk}^L}  =  \delta_j^La_k^{L-1}.
+\frac{\partial{\cal C}}{\partial w_{ij}^L}  =  \delta_j^La_i^{L-1}.
 \]
 !et
 
@@ -749,7 +750,7 @@ We have now three equations that are essential for the computations of the deriv
 
 !bt
 \begin{equation}
-\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{ij}^L}  =  \delta_j^La_i^{L-1},
+\frac{\partial{\cal C}(\bm{W^L})}{\partial w_{ij}^L}  =  \delta_j^La_i^{L-1},
 \end{equation}
 !et
 and
@@ -808,13 +809,13 @@ We are now ready to set up the algorithm for back propagation and learning the w
 
 The four equations  provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
 
-_First_, we set up the input data $\hat{x}$ and the activations
-$\hat{z}_1$ of the input layer and compute the activation function and
-the pertinent outputs $\hat{a}^1$.
+_First_, we set up the input data $\bm{x}$ and the activations
+$\bm{z}_1$ of the input layer and compute the activation function and
+the pertinent outputs $\bm{a}^1$.
 
 _Secondly_, we perform then the feed forward till we reach the output
-layer and compute all $\hat{z}_l$ of the input layer and compute the
-activation function and the pertinent outputs $\hat{a}^l$ for
+layer and compute all $\bm{z}_l$ of the input layer and compute the
+activation function and the pertinent outputs $\bm{a}^l$ for
 $l=1,2,3,\dots,L$.
 
 
@@ -824,7 +825,7 @@ _Notation_: The first hidden layer has $l=1$ as label and the final output layer
 ===== Setting up the back propagation algorithm, part 2 =====
 
 
-Thereafter we compute the ouput error $\hat{\delta}^L$ by computing all
+Thereafter we compute the ouput error $\bm{\delta}^L$ by computing all
 !bt
 \[
 \delta_j^L = \sigma'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}.
@@ -843,12 +844,12 @@ Then we compute the back propagate error for each $l=L-1,L-2,\dots,1$ as
 
 
 Finally, we update the weights and the biases using gradient descent
-for each $l=L-1,L-2,\dots,1$ and update the weights and biases
+for each $l=L-1,L-2,\dots,1$ (the first hidden layer) and update the weights and biases
 according to the rules
 
 !bt
 \[
-w_{jk}^l\leftarrow  = w_{jk}^l- \eta \delta_j^la_k^{l-1},
+w_{ij}^l\leftarrow  = w_{ij}^l- \eta \delta_j^la_i^{l-1},
 \]
 !et
 
@@ -865,13 +866,13 @@ with $\eta$ being the learning rate.
 With the back propagate error for each $l=L-1,L-2,\dots,1$ as
 !bt
 \[
-\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}sigma'(z_j^l),
+\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l),
 \]
 !et
 we update the weights and the biases using gradient descent for each $l=L-1,L-2,\dots,1$ and update the weights and biases according to the rules
 !bt
 \[
-w_{jk}^l\leftarrow  = w_{jk}^l- \eta \delta_j^la_k^{l-1},
+w_{ij}^l\leftarrow  = w_{ij}^l- \eta \delta_j^la_i^{l-1},
 \]
 !et
 
@@ -917,13 +918,13 @@ functions Typical examples are the logistic *Sigmoid*
 
 !bt
 \[
- f(x) = \frac{1}{1 + e^{-x}},
+ \sigma(x) = \frac{1}{1 + e^{-x}},
 \]
 !et
 and the *hyperbolic tangent* function
 !bt
 \[
- f(x) = \tanh(x)
+ \sigma(x) = \tanh(x)
 \]
 !et
 
@@ -1012,10 +1013,6 @@ plt.show()
 
 
 
-
-
-
-
 !split
 ===== Fine-tuning neural network hyperparameters =====
 
@@ -1378,7 +1375,7 @@ Thereafter we compute the ouput error $\bm{\delta}^L$ by computing all
 Then we compute the back propagate error for each $l=L-1,L-2,\dots,2$ as
 !bt
 \[
-\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).
+\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l).
 \]
 !et
 !eblock
@@ -1387,7 +1384,7 @@ Then we compute the back propagate error for each $l=L-1,L-2,\dots,2$ as
 Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\dots,2$ and update the weights and biases according to the rules
 !bt
 \[
-w_{jk}^l\leftarrow  = w_{jk}^l- \eta \delta_j^la_k^{l-1},
+w_{ij}^l\leftarrow  = w_{ij}^l- \eta \delta_j^la_i^{l-1},
 \]
 !et