-
+One use often so-called fully-connected feed-forward neural networks with three +or more layers (an input layer, one or more hidden layers and an output layer) +consisting of neurons that have non-linear activation functions. + +
+Such networks are often called multilayer perceptrons (MLPs) + +
+ + +
+According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing +a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, +assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function. +Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always +assumed to be linear, so as to not restrict the range of output values. + +
+We note that this theorem is only applicable to a NN with one hidden layer. +Therefore, we can easily construct an NN +that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer +with activation functions that do. Furthermore, although the universal approximation theorem +lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: +A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other +functions. + +
+ + +
-
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \), @@ -336,7 +385,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
-
The output of neuron \( i \) in layer 2 is thus, @@ -365,7 +414,7 @@ $$
-
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form @@ -385,7 +434,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
-
This confirms that an MLP, @@ -407,19 +456,12 @@ $$ where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions can be shifted up and down or left and right, change slope or be rescaled -which is the key to the flexibility of a NN. - -$$ -\begin{equation} - f_o = f(u_o) = u_o -\label{outputActivation} -\end{equation} -$$ +which is the key to the flexibility of a neural network.
-
+ + +
+ + +
+A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). +As described in, the following restrictions are imposed on an activation function for a FFNN +to fulfill the universal approximation theorem + +
+The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each +layer simply performs a linear transformation of its inputs. + +
+Regardless of the number of layers, +the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of +non-linearity to the NN to be able to fit non-linear functions +Typical examples are the logistic Sigmoid +$$ +\begin{equation} + f(x) = \frac{1}{1 + e^{-x}}, +\label{sigmoidActivationFunction} +\end{equation} +$$ + +and the hyperbolic tangent function +$$ +\begin{equation} + f(x) = \tanh(x) +\label{tanhActivationFunction} +\end{equation} +$$ + +
+ + +
diff --git a/doc/pub/NeuralNet/html/NeuralNet-reveal.html b/doc/pub/NeuralNet/html/NeuralNet-reveal.html
index 6e4dcb45b..84cc3cc01 100644
--- a/doc/pub/NeuralNet/html/NeuralNet-reveal.html
+++ b/doc/pub/NeuralNet/html/NeuralNet-reveal.html
@@ -304,7 +304,41 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
+One use often so-called fully-connected feed-forward neural networks with three
+or more layers (an input layer, one or more hidden layers and an output layer)
+consisting of neurons that have non-linear activation functions.
+
+
+Such networks are often called multilayer perceptrons (MLPs)
+
+According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing
+a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
+assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function.
+Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
+assumed to be linear, so as to not restrict the range of output values.
+
+
+We note that this theorem is only applicable to a NN with one hidden layer.
+Therefore, we can easily construct an NN
+that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
+with activation functions that do. Furthermore, although the universal approximation theorem
+lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
+A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other
+functions.
+
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -365,7 +399,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
The output of neuron \( i \) in layer 2 is thus,
@@ -398,7 +432,7 @@ $$
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -420,7 +454,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
This confirms that an MLP,
@@ -444,22 +478,13 @@ $$
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
-which is the key to the flexibility of a NN.
-
-
+A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
+As described in, the following restrictions are imposed on an activation function for a FFNN
+to fulfill the universal approximation theorem
+
+
+The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
+layer simply performs a linear transformation of its inputs.
+
+
+Regardless of the number of layers,
+the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
+non-linearity to the NN to be able to fit non-linear functions
+Typical examples are the logistic Sigmoid
+
+One use often so-called fully-connected feed-forward neural networks with three
+or more layers (an input layer, one or more hidden layers and an output layer)
+consisting of neurons that have non-linear activation functions.
+
+
+Such networks are often called multilayer perceptrons (MLPs)
+
+
+
+According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing
+a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
+assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function.
+Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
+assumed to be linear, so as to not restrict the range of output values.
+
+
+We note that this theorem is only applicable to a NN with one hidden layer.
+Therefore, we can easily construct an NN
+that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
+with activation functions that do. Furthermore, although the universal approximation theorem
+lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
+A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other
+functions.
+
+
+
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -285,7 +328,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
The output of neuron \( i \) in layer 2 is thus,
@@ -314,7 +357,7 @@ $$
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -334,7 +377,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
This confirms that an MLP,
@@ -356,19 +399,12 @@ $$
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
-which is the key to the flexibility of a NN.
-
-$$
-\begin{equation}
- f_o = f(u_o) = u_o
-\label{outputActivation}
-\end{equation}
-$$
+which is the key to the flexibility of a neural network.
+
+
+A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
+As described in, the following restrictions are imposed on an activation function for a FFNN
+to fulfill the universal approximation theorem
+
+
+The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
+layer simply performs a linear transformation of its inputs.
+
+
+Regardless of the number of layers,
+the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
+non-linearity to the NN to be able to fit non-linear functions
+Typical examples are the logistic Sigmoid
+$$
+\begin{equation}
+ f(x) = \frac{1}{1 + e^{-x}},
+\label{sigmoidActivationFunction}
+\end{equation}
+$$
+
+and the hyperbolic tangent function
+$$
+\begin{equation}
+ f(x) = \tanh(x)
+\label{tanhActivationFunction}
+\end{equation}
+$$
+
+
+
diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html
index 04804c9c2..c43db64cc 100644
--- a/doc/pub/NeuralNet/html/NeuralNet.html
+++ b/doc/pub/NeuralNet/html/NeuralNet.html
@@ -46,12 +46,21 @@ div { text-align: justify; text-justify: inter-word; }
('Feed-forward neural networks', 2, None, '___sec4'),
('Recurrent neural networks', 2, None, '___sec5'),
('Other types of networks', 2, None, '___sec6'),
- ('Mathematical model', 2, None, '___sec7'),
- ('Mathematical model', 2, None, '___sec8'),
+ ('Multilayer perceptrons', 2, None, '___sec7'),
+ ('Why multilayer perceptrons?', 2, None, '___sec8'),
('Mathematical model', 2, None, '___sec9'),
('Mathematical model', 2, None, '___sec10'),
('Mathematical model', 2, None, '___sec11'),
- ('Matrix-vector notation', 3, None, '___sec12')]}
+ ('Mathematical model', 2, None, '___sec12'),
+ ('Mathematical model', 2, None, '___sec13'),
+ ('Matrix-vector notation', 3, None, '___sec14'),
+ ('Matrix-vector notation and activation', 3, None, '___sec15'),
+ ('Activation functions', 3, None, '___sec16'),
+ ('Activation functions, Logistic and Hyperbolic ones',
+ 3,
+ None,
+ '___sec17'),
+ ('Relevance', 3, None, '___sec18')]}
end of tocinfo -->
+One use often so-called fully-connected feed-forward neural networks with three
+or more layers (an input layer, one or more hidden layers and an output layer)
+consisting of neurons that have non-linear activation functions.
+
+
+Such networks are often called multilayer perceptrons (MLPs)
+
+
+
+According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing
+a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
+assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function.
+Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
+assumed to be linear, so as to not restrict the range of output values.
+
+
+We note that this theorem is only applicable to a NN with one hidden layer.
+Therefore, we can easily construct an NN
+that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
+with activation functions that do. Furthermore, although the universal approximation theorem
+lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
+A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other
+functions.
+
+
+
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -290,7 +333,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
The output of neuron \( i \) in layer 2 is thus,
@@ -319,7 +362,7 @@ $$
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -339,7 +382,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
This confirms that an MLP,
@@ -361,19 +404,12 @@ $$
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
-which is the key to the flexibility of a NN.
-
-$$
-\begin{equation}
- f_o = f(u_o) = u_o
-\label{outputActivation}
-\end{equation}
-$$
+which is the key to the flexibility of a neural network.
+
+
+A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
+As described in, the following restrictions are imposed on an activation function for a FFNN
+to fulfill the universal approximation theorem
+
+
+The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
+layer simply performs a linear transformation of its inputs.
+
+
+Regardless of the number of layers,
+the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
+non-linearity to the NN to be able to fit non-linear functions
+Typical examples are the logistic Sigmoid
+$$
+\begin{equation}
+ f(x) = \frac{1}{1 + e^{-x}},
+\label{sigmoidActivationFunction}
+\end{equation}
+$$
+
+and the hyperbolic tangent function
+$$
+\begin{equation}
+ f(x) = \tanh(x)
+\label{tanhActivationFunction}
+\end{equation}
+$$
+
+
+
diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
index 3b424cdc9..ab5655a33 100644
--- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
+++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
@@ -142,6 +142,30 @@
"Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description\n",
"of how a fully-connected FFNN works, and how it can be used to interpolate data sets. \n",
"\n",
+ "## Multilayer perceptrons\n",
+ "\n",
+ "One use often so-called fully-connected feed-forward neural networks with three\n",
+ "or more layers (an input layer, one or more hidden layers and an output layer)\n",
+ "consisting of neurons that have non-linear activation functions.\n",
+ "\n",
+ "Such networks are often called *multilayer perceptrons* (MLPs)\n",
+ "\n",
+ "## Why multilayer perceptrons?\n",
+ "\n",
+ "According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing \n",
+ "a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, \n",
+ "assuming the activation function for the hidden layer is a **non-constant, bounded and monotonically-increasing continuous function**.\n",
+ "Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always\n",
+ "assumed to be linear, so as to not restrict the range of output values. \n",
+ "\n",
+ "We note that this theorem is only applicable to a NN with *one* hidden layer. \n",
+ "Therefore, we can easily construct an NN \n",
+ "that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer\n",
+ "with activation functions that *do*. Furthermore, although the universal approximation theorem\n",
+ "lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: \n",
+ "A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit *all other*\n",
+ "functions. \n",
+ "\n",
"\n",
"\n",
"## Mathematical model"
@@ -167,7 +191,7 @@
"metadata": {},
"source": [
"In an FFNN of such neurons, the *inputs* $x_i$\n",
- "are the *outputs* of the neurons in the preceding layer. Furthermore, a MLP is fully-connected, \n",
+ "are the *outputs* of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, \n",
"which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer. \n",
"\n",
"## Mathematical model\n",
@@ -384,28 +408,8 @@
"source": [
"where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions\n",
"can be shifted up and down or left and right, change slope or be rescaled \n",
- "which is the key to the flexibility of a NN."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "Mathematical model
+Multilayer perceptrons
+
+Why multilayer perceptrons?
+
+Mathematical model
$$
@@ -316,13 +350,13 @@ $$
In an FFNN of such neurons, the inputs \( x_i \)
-are the outputs of the neurons in the preceding layer. Furthermore, a MLP is fully-connected,
+are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer.
Mathematical model
+Mathematical model
Mathematical model
+Mathematical model
Mathematical model
+Mathematical model
Mathematical model
+Mathematical model
-$$
-\begin{equation}
- f_o = f(u_o) = u_o
-\tag{12}
-\end{equation}
-$$
-
+which is the key to the flexibility of a neural network.
Matrix-vector notation
+Matrix-vector notation
We can introduce a more convenient notation for the activations in a NN.
@@ -472,7 +497,6 @@ is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l
We have that \( \mathrm{W}_l \) is a \( N_{l-1} \times N_l \) matrix, while \( \vec{b}_l \) and \( \vec{y}_l \) are \( N_l \times 1 \) column vectors.
With this notation, the sum in becomes a matrix-vector multiplication, and we can write
the equation for the activations of hidden layer 2 in
-
$$
\begin{equation}
@@ -491,29 +515,102 @@ $$
b^2_1 \\
b^2_2 \\
b^2_3 \\
- \end{array}\right]\right)
-\tag{13}
+ \end{array}\right]\right).
+\tag{12}
\end{equation}
$$
+Matrix-vector notation and activation
+
+The activation of node \( i \) in layer 2 is
$$
\begin{equation}
y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
- f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right)
+ f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
+\tag{13}
+\end{equation}
+$$
+
+
+This is not just a convenient and compact notation, but also
+a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
+and vector additions that are used as input to the activation functions. For each operation
+\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
+Activation functions
+
+
+
+
+Activation functions, Logistic and Hyperbolic ones
+
+
+$$
+\begin{equation}
+ f(x) = \frac{1}{1 + e^{-x}},
\tag{14}
\end{equation}
$$
-which is in accordance with. Note that
-This is not just a convenient and compact notation, but also
-a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
-and vector additions that are used as input to the activation functions. For each operation
-\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
+and the hyperbolic tangent function
+
+$$
+\begin{equation}
+ f(x) = \tanh(x)
+\tag{15}
+\end{equation}
+$$
+
+Relevance
+
+The sigmoid function are more biologically plausible because
+the output of inactive neurons are zero. Such activation function are called one-sided. However,
+it has been shown that the hyperbolic tangent
+performs better than the sigmoid for training MLPs.
+has become the most popular for deep neural networks
-Mathematical model
+Multilayer perceptrons
+
+
+
+Why multilayer perceptrons?
+
+
+
+Mathematical model
$$
\begin{equation}
@@ -242,13 +285,13 @@ $$
$$
In an FFNN of such neurons, the inputs \( x_i \)
-are the outputs of the neurons in the preceding layer. Furthermore, a MLP is fully-connected,
+are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer.
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Matrix-vector notation
+Matrix-vector notation
We can introduce a more convenient notation for the activations in a NN.
@@ -381,7 +417,6 @@ is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l
We have that \( \mathrm{W}_l \) is a \( N_{l-1} \times N_l \) matrix, while \( \vec{b}_l \) and \( \vec{y}_l \) are \( N_l \times 1 \) column vectors.
With this notation, the sum in becomes a matrix-vector multiplication, and we can write
the equation for the activations of hidden layer 2 in
-
$$
\begin{equation}
\vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) =
@@ -399,27 +434,87 @@ $$
b^2_1 \\
b^2_2 \\
b^2_3 \\
- \end{array}\right]\right)
+ \end{array}\right]\right).
\label{_auto6}
\end{equation}
$$
-and we see that the activation of node \( i \) in layer 2 is
+
+
+Matrix-vector notation and activation
+
+The activation of node \( i \) in layer 2 is
$$
\begin{equation}
y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
- f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right)
+ f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
\label{_auto7}
\end{equation}
$$
-which is in accordance with. Note that
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
+
+
+Activation functions
+
+
+
+
+
+
+Activation functions, Logistic and Hyperbolic ones
+
+
+
+Relevance
+
+The sigmoid function are more biologically plausible because
+the output of inactive neurons are zero. Such activation function are called one-sided. However,
+it has been shown that the hyperbolic tangent
+performs better than the sigmoid for training MLPs.
+has become the most popular for deep neural networks
+
-Mathematical model
+Multilayer perceptrons
+
+
+
+Why multilayer perceptrons?
+
+
+
+Mathematical model
$$
\begin{equation}
@@ -247,13 +290,13 @@ $$
$$
In an FFNN of such neurons, the inputs \( x_i \)
-are the outputs of the neurons in the preceding layer. Furthermore, a MLP is fully-connected,
+are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer.
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Matrix-vector notation
+Matrix-vector notation
We can introduce a more convenient notation for the activations in a NN.
@@ -386,7 +422,6 @@ is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l
We have that \( \mathrm{W}_l \) is a \( N_{l-1} \times N_l \) matrix, while \( \vec{b}_l \) and \( \vec{y}_l \) are \( N_l \times 1 \) column vectors.
With this notation, the sum in becomes a matrix-vector multiplication, and we can write
the equation for the activations of hidden layer 2 in
-
$$
\begin{equation}
\vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) =
@@ -404,27 +439,87 @@ $$
b^2_1 \\
b^2_2 \\
b^2_3 \\
- \end{array}\right]\right)
+ \end{array}\right]\right).
\label{_auto6}
\end{equation}
$$
-and we see that the activation of node \( i \) in layer 2 is
+
+
+Matrix-vector notation and activation
+
+The activation of node \( i \) in layer 2 is
$$
\begin{equation}
y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
- f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right)
+ f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
\label{_auto7}
\end{equation}
$$
-which is in accordance with. Note that
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
+
+
+Activation functions
+
+
+
+
+
+
+Activation functions, Logistic and Hyperbolic ones
+
+
+
+Relevance
+
+The sigmoid function are more biologically plausible because
+the output of inactive neurons are zero. Such activation function are called one-sided. However,
+it has been shown that the hyperbolic tangent
+performs better than the sigmoid for training MLPs.
+has become the most popular for deep neural networks
+