small update on neural networks

doc/src/NeuralNet/NeuralNet.do.txt

@@ -1007,6 +1007,29 @@ where we have defined the targets $t_i$. The derivatives of the cost function wi
 In case we use another activation function than the logistic one, we need to evaluate other derivatives. 
 
 
+!split
+===== The Softmax function =====
+In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need
+!bt
+\[
+\frac{\partial f(z_i^l)}{\partial w_{jk}^l} =
+\frac{\partial f(z_i^l)}{\partial z_j^l} \frac{\partial z_j^l}{\partial w_{jk}^l}= \frac{\partial f(z_i^l)}{\partial z_j^l}a_k^{-1}.
+\]
+!et
+For the Softmax function we have
+!bt
+\[
+f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{k=1}^K\exp{(z_k^l}}.
+\]
+!et
+Its derivative with respect to $z_j^l$ gives 
+!bt
+\[
+\frac{\partial f(z_i^l)}{\partial z_j^l}= f(z_i^l)\left(\delta_{ij}-f(z_i^l)\right), 
+\]
+!et
+which in case of the simply binary model reduces to  having $i=j$. 
+
 !split 
 ===== Developing a code for doing neural networks with back propagation =====