Update on NN

doc/src/NeuralNet/NeuralNet.do.txt

@@ -122,6 +122,32 @@ a separate type of NN due the unusual activation functions.
 Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
 of how a fully-connected FFNN works, and how it can be used to interpolate data sets. 
 
+!split
+===== Multilayer perceptrons  =====
+
+One use often so-called  fully-connected feed-forward neural networks with three
+or more layers (an input layer, one or more hidden layers and an output layer)
+consisting of neurons that have non-linear activation functions.
+
+Such networks are often called *multilayer perceptrons* (MLPs)
+
+!split
+===== Why multilayer perceptrons?  =====
+
+According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing 
+a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, 
+assuming the activation function for the hidden layer is a _non-constant, bounded and monotonically-increasing continuous function_.
+Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
+assumed to be linear, so as to not restrict the range of output values. 
+
+We note that this theorem is only applicable to a NN with *one* hidden layer. 
+Therefore, we can easily construct an NN 
+that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
+with activation functions that *do*. Furthermore, although the universal approximation theorem
+lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: 
+A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit *all other*
+functions. 
+
 
 
 !split
@@ -134,7 +160,7 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
 \end{equation}
 !et
 In an FFNN of such neurons, the *inputs* $x_i$
-are the *outputs* of the neurons in the preceding layer. Furthermore, a MLP is fully-connected, 
+are the *outputs* of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, 
 which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer. 
 
 !split
@@ -226,16 +252,7 @@ the expression  is essentially a nested sum of scaled activation functions of th
 !et
 where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions
 can be shifted up and down or left and right, change slope or be rescaled 
-which is the key to the flexibility of a NN. 
-
-
-!bt
-\begin{equation}
- f_o = f(u_o) = u_o
- label{outputActivation}
-\end{equation}
-!et
-
+which is the key to the flexibility of a neural network. 
 
 !split
 === Matrix-vector notation ===
@@ -248,7 +265,6 @@ is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively
 We have that $\mathrm{W}_l$ is a $N_{l-1} \times N_l$ matrix, while $\vec{b}_l$ and $\vec{y}_l$ are $N_l \times 1$ column vectors. 
 With this notation, the sum in  becomes a matrix-vector multiplication, and we can write
 the equation for the activations of hidden layer 2 in
-
 !bt
 \begin{equation}
  \vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) = 
@@ -266,21 +282,72 @@ the equation for the activations of hidden layer 2 in
            b^2_1 \\
            b^2_2 \\
            b^2_3 \\
-          \end{array}\right]\right)
+          \end{array}\right]\right).
 \end{equation}
 !et
-and we see that the activation of node $i$ in layer 2 is
+
+!split
+=== Matrix-vector notation  and activation ===
+The activation of node $i$ in layer 2 is
 
 !bt
 \begin{equation}
  y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) = 
- f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right)
+ f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
 \end{equation}
 !et 
-which is in accordance with. Note that
 This is not just a convenient and compact notation, but also 
 a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
 and vector additions that are used as input to the activation functions. For each operation 
 $\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer. 
 
 
+!split
+=== Activation functions  ===
+
+
+A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). 
+As described in, the following restrictions are imposed on an activation function for a FFNN
+to fulfill the universal approximation theorem
+
+  * Non-constant
+
+  * Bounded
+
+  * Monotonically-increasing
+
+  * Continuous
+
+!split
+=== Activation functions, Logistic and Hyperbolic ones  ===
+
+The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each 
+layer simply performs a linear transformation of its inputs.
+
+Regardless of the number of layers, 
+the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of 
+non-linearity to the NN to be able to fit non-linear functions
+Typical examples are the logistic *Sigmoid*
+!bt
+\begin{equation}
+ f(x) = \frac{1}{1 + e^{-x}},
+ label{sigmoidActivationFunction}
+\end{equation}
+!et
+and the *hyperbolic tangent* function
+!bt
+\begin{equation}
+ f(x) = \tanh(x)
+ label{tanhActivationFunction}
+\end{equation}
+!et
+
+!split
+=== Relevance === 
+The *sigmoid* function are more biologically plausible because 
+the output of inactive neurons are zero. Such activation function are called *one-sided*. However,
+it has been shown  that the hyperbolic tangent 
+performs better than the sigmoid for training MLPs. 
+has become the most popular  for *deep neural networks*
+
+