-Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description -of how a fully-connected FFNN works, and how it can be used to interpolate data sets. -
-One use often so-called fully-connected feed-forward neural networks with three -or more layers (an input layer, one or more hidden layers and an output layer) -consisting of neurons that have non-linear activation functions. +One uses often so-called fully-connected feed-forward neural networks +with three or more layers (an input layer, one or more hidden layers +and an output layer) consisting of neurons that have non-linear +activation functions.
Such networks are often called multilayer perceptrons (MLPs) @@ -348,17 +345,22 @@ Such networks are often called multilayer perceptrons (MLPs) According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function. -Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always -assumed to be linear, so as to not restrict the range of output values.
-We note that this theorem is only applicable to a NN with one hidden layer. -Therefore, we can easily construct an NN -that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer -with activation functions that do. Furthermore, although the universal approximation theorem -lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: -A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other -functions. +Note that the requirements on the activation function only applies to +the hidden layer, the output nodes are always assumed to be linear, so +as to not restrict the range of output values. + +
+We note that this theorem is only applicable to an NN with one hidden +layer. Therefore, we can easily construct an NN that employs +activation functions which do not satisfy the above requirements, as +long as we have at least one layer with activation functions that +do. Furthermore, although the universal approximation theorem lays +the theoretical foundation for regression with neural networks, it +does not say anything about how things work in practice: A neural +network can still be able to approximate a given function reasonably +well without having the flexibility to fit all other functions.
@@ -372,9 +374,11 @@ $$ \end{equation} $$ -In an FFNN of such neurons, the inputs \( x_i \) -are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, -which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer. +
+In an FFNN of such neurons, the inputs \( x_i \) are the outputs of +the neurons in the preceding layer. Furthermore, an MLP is +fully-connected, which means that each neuron receives a weighted sum +of the outputs of all neurons in the previous layer.
@@ -401,7 +405,9 @@ $$ \end{equation} $$ -where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \) +
+where we assume that all nodes in the same layer have identical +activation functions, hence the notation \( f_l \) $$ \begin{equation} @@ -410,8 +416,11 @@ $$ \end{equation} $$ -where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed, -the values of the subsequent layer can be calculated and so forth until the output is obtained. +
+where \( N_l \) is the number of nodes in layer \( l \). When the output of +all the nodes in the first hidden layer are computed, the values of +the subsequent layer can be calculated and so forth until the output +is obtained.
@@ -448,8 +457,9 @@ $$
-We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form -is, +We can generalize this expression to an MLP with \( l \) hidden +layers. The complete functional form is, + $$ \begin{align} &y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\! @@ -460,7 +470,9 @@ $$ \end{align} $$ -which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \). +
+which illustrates a basic property of MLPs: The only independent +variables are the input values \( x_n \).
@@ -468,15 +480,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
-This confirms that an MLP, -despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a -mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \). -In our example, \( n=2 \) and \( m=1 \). Consequentially, -the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. +This confirms that an MLP, despite its quite convoluted mathematical +form, is nothing more than an analytic function, specifically a +mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow +\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and +\( m=1 \). Consequentially, the number of input and output values of the +function we want to fit must be equal to the number of inputs and +outputs of our MLP.
-Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that -the expression is essentially a nested sum of scaled activation functions of the form +Furthermore, the flexibility and universality of a MLP can be +illustrated by realizing that the expression is essentially a nested +sum of scaled activation functions of the form $$ \begin{equation} @@ -485,15 +500,18 @@ $$ \end{equation} $$ -where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions -can be shifted up and down or left and right, change slope or be rescaled -which is the key to the flexibility of a neural network. +
+where the parameters \( c_i \) are weights and biases. By adjusting these +parameters, the activation functions can be shifted up and down or +left and right, change slope or be rescaled which is the key to the +flexibility of a neural network.
We can introduce a more convenient notation for the activations in a NN.
@@ -532,6 +550,7 @@ $$
The activation of node \( i \) in layer 2 is $$ @@ -542,9 +561,11 @@ $$ \end{equation} $$ -This is not just a convenient and compact notation, but also -a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications -and vector additions that are used as input to the activation functions. For each operation +
+This is not just a convenient and compact notation, but also a useful +and intuitive way to think about MLPs: The output is calculated by a +series of matrix-vector multiplications and vector additions that are +used as input to the activation functions. For each operation \( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
@@ -553,9 +574,10 @@ and vector additions that are used as input to the activation functions. For eac
-A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). -As described in, the following restrictions are imposed on an activation function for a FFNN -to fulfill the universal approximation theorem +A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). As described +in, the following restrictions are imposed on an activation function +for a FFNN to fulfill the universal approximation theorem
-The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each -layer simply performs a linear transformation of its inputs. +The second requirement excludes all linear functions. Furthermore, in +a MLP with only linear activation functions, each layer simply +performs a linear transformation of its inputs.
-Regardless of the number of layers, -the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of -non-linearity to the NN to be able to fit non-linear functions -Typical examples are the logistic Sigmoid +Regardless of the number of layers, the output of the NN will be +nothing but a linear function of the inputs. Thus we need to introduce +some kind of non-linearity to the NN to be able to fit non-linear +functions Typical examples are the logistic Sigmoid + $$ \begin{equation} f(x) = \frac{1}{1 + e^{-x}}, @@ -597,11 +621,12 @@ $$
+The sigmoid function are more biologically plausible because the +output of inactive neurons are zero. Such activation function are +called one-sided. However, it has been shown that the hyperbolic +tangent performs better than the sigmoid for training MLPs. has +become the most popular for deep neural networks
diff --git a/doc/pub/NeuralNet/html/NeuralNet-reveal.html b/doc/pub/NeuralNet/html/NeuralNet-reveal.html index 99cb9c0d8..53e61e540 100644 --- a/doc/pub/NeuralNet/html/NeuralNet-reveal.html +++ b/doc/pub/NeuralNet/html/NeuralNet-reveal.html @@ -319,10 +319,6 @@ an input layer, a hidden layer with non-linear radial symmetric activation funct means that each node in the output layer has a linear activation function). The layers are normally fully-connected and there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as a separate type of NN due the unusual activation functions. - -
-Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description -of how a fully-connected FFNN works, and how it can be used to interpolate data sets. @@ -330,9 +326,10 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
-One use often so-called fully-connected feed-forward neural networks with three -or more layers (an input layer, one or more hidden layers and an output layer) -consisting of neurons that have non-linear activation functions. +One uses often so-called fully-connected feed-forward neural networks +with three or more layers (an input layer, one or more hidden layers +and an output layer) consisting of neurons that have non-linear +activation functions.
Such networks are often called multilayer perceptrons (MLPs) @@ -346,17 +343,22 @@ Such networks are often called multilayer perceptrons (MLPs) According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function. -Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always -assumed to be linear, so as to not restrict the range of output values.
-We note that this theorem is only applicable to a NN with one hidden layer. -Therefore, we can easily construct an NN -that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer -with activation functions that do. Furthermore, although the universal approximation theorem -lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: -A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other -functions. +Note that the requirements on the activation function only applies to +the hidden layer, the output nodes are always assumed to be linear, so +as to not restrict the range of output values. + +
+We note that this theorem is only applicable to an NN with one hidden +layer. Therefore, we can easily construct an NN that employs +activation functions which do not satisfy the above requirements, as +long as we have at least one layer with activation functions that +do. Furthermore, although the universal approximation theorem lays +the theoretical foundation for regression with neural networks, it +does not say anything about how things work in practice: A neural +network can still be able to approximate a given function reasonably +well without having the flexibility to fit all other functions. @@ -372,9 +374,11 @@ $$ $$
-In an FFNN of such neurons, the inputs \( x_i \)
-are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
-which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer.
+
+In an FFNN of such neurons, the inputs \( x_i \) are the outputs of +the neurons in the preceding layer. Furthermore, an MLP is +fully-connected, which means that each neuron receives a weighted sum +of the outputs of all neurons in the previous layer. @@ -405,7 +409,9 @@ $$ $$
-where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
+
+where we assume that all nodes in the same layer have identical +activation functions, hence the notation \( f_l \)
$$
@@ -416,8 +422,11 @@ $$
$$
-where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed,
-the values of the subsequent layer can be calculated and so forth until the output is obtained.
+
+where \( N_l \) is the number of nodes in layer \( l \). When the output of +all the nodes in the first hidden layer are computed, the values of +the subsequent layer can be calculated and so forth until the output +is obtained. @@ -458,8 +467,9 @@ $$
-We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form -is, +We can generalize this expression to an MLP with \( l \) hidden +layers. The complete functional form is, +
$$
\begin{align}
@@ -472,7 +482,9 @@ $$
$$
-which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \).
+
+which illustrates a basic property of MLPs: The only independent +variables are the input values \( x_n \). @@ -480,15 +492,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
-This confirms that an MLP, -despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a -mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \). -In our example, \( n=2 \) and \( m=1 \). Consequentially, -the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. +This confirms that an MLP, despite its quite convoluted mathematical +form, is nothing more than an analytic function, specifically a +mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow +\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and +\( m=1 \). Consequentially, the number of input and output values of the +function we want to fit must be equal to the number of inputs and +outputs of our MLP.
-Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that -the expression is essentially a nested sum of scaled activation functions of the form +Furthermore, the flexibility and universality of a MLP can be +illustrated by realizing that the expression is essentially a nested +sum of scaled activation functions of the form
$$
@@ -499,9 +514,11 @@ $$
$$
-where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
-can be shifted up and down or left and right, change slope or be rescaled
-which is the key to the flexibility of a neural network.
+
+where the parameters \( c_i \) are weights and biases. By adjusting these +parameters, the activation functions can be shifted up and down or +left and right, change slope or be rescaled which is the key to the +flexibility of a neural network. @@ -509,6 +526,7 @@ which is the key to the flexibility of a neural network.
We can introduce a more convenient notation for the activations in a NN.
@@ -550,6 +568,7 @@ $$
The activation of node \( i \) in layer 2 is
@@ -562,9 +581,11 @@ $$
$$
-This is not just a convenient and compact notation, but also
-a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
-and vector additions that are used as input to the activation functions. For each operation
+
+This is not just a convenient and compact notation, but also a useful +and intuitive way to think about MLPs: The output is calculated by a +series of matrix-vector multiplications and vector additions that are +used as input to the activation functions. For each operation \( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer. @@ -574,9 +595,10 @@ and vector additions that are used as input to the activation functions. For eac
-A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). -As described in, the following restrictions are imposed on an activation function for a FFNN -to fulfill the universal approximation theorem +A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). As described +in, the following restrictions are imposed on an activation function +for a FFNN to fulfill the universal approximation theorem
-The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each -layer simply performs a linear transformation of its inputs. +The second requirement excludes all linear functions. Furthermore, in +a MLP with only linear activation functions, each layer simply +performs a linear transformation of its inputs.
-Regardless of the number of layers, -the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of -non-linearity to the NN to be able to fit non-linear functions -Typical examples are the logistic Sigmoid +Regardless of the number of layers, the output of the NN will be +nothing but a linear function of the inputs. Thus we need to introduce +some kind of non-linearity to the NN to be able to fit non-linear +functions Typical examples are the logistic Sigmoid +
$$
\begin{equation}
@@ -629,11 +653,12 @@ $$
+The sigmoid function are more biologically plausible because the +output of inactive neurons are zero. Such activation function are +called one-sided. However, it has been shown that the hyperbolic +tangent performs better than the sigmoid for training MLPs. has +become the most popular for deep neural networks
diff --git a/doc/pub/NeuralNet/html/NeuralNet-solarized.html b/doc/pub/NeuralNet/html/NeuralNet-solarized.html index eccecb928..6dbebedba 100644 --- a/doc/pub/NeuralNet/html/NeuralNet-solarized.html +++ b/doc/pub/NeuralNet/html/NeuralNet-solarized.html @@ -264,19 +264,16 @@ means that each node in the output layer has a linear activation function). The there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as a separate type of NN due the unusual activation functions. -
-Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description -of how a fully-connected FFNN works, and how it can be used to interpolate data sets. -
-One use often so-called fully-connected feed-forward neural networks with three -or more layers (an input layer, one or more hidden layers and an output layer) -consisting of neurons that have non-linear activation functions. +One uses often so-called fully-connected feed-forward neural networks +with three or more layers (an input layer, one or more hidden layers +and an output layer) consisting of neurons that have non-linear +activation functions.
Such networks are often called multilayer perceptrons (MLPs) @@ -290,17 +287,22 @@ Such networks are often called multilayer perceptrons (MLPs) According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function. -Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always -assumed to be linear, so as to not restrict the range of output values.
-We note that this theorem is only applicable to a NN with one hidden layer. -Therefore, we can easily construct an NN -that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer -with activation functions that do. Furthermore, although the universal approximation theorem -lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: -A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other -functions. +Note that the requirements on the activation function only applies to +the hidden layer, the output nodes are always assumed to be linear, so +as to not restrict the range of output values. + +
+We note that this theorem is only applicable to an NN with one hidden +layer. Therefore, we can easily construct an NN that employs +activation functions which do not satisfy the above requirements, as +long as we have at least one layer with activation functions that +do. Furthermore, although the universal approximation theorem lays +the theoretical foundation for regression with neural networks, it +does not say anything about how things work in practice: A neural +network can still be able to approximate a given function reasonably +well without having the flexibility to fit all other functions.
@@ -314,9 +316,11 @@ $$
\end{equation}
$$
-In an FFNN of such neurons, the inputs \( x_i \)
-are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
-which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer.
+
+In an FFNN of such neurons, the inputs \( x_i \) are the outputs of +the neurons in the preceding layer. Furthermore, an MLP is +fully-connected, which means that each neuron receives a weighted sum +of the outputs of all neurons in the previous layer.
@@ -343,7 +347,9 @@ $$
\end{equation}
$$
-where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
+
+where we assume that all nodes in the same layer have identical +activation functions, hence the notation \( f_l \) $$ \begin{equation} @@ -352,8 +358,11 @@ $$ \end{equation} $$ -where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed, -the values of the subsequent layer can be calculated and so forth until the output is obtained. +
+where \( N_l \) is the number of nodes in layer \( l \). When the output of +all the nodes in the first hidden layer are computed, the values of +the subsequent layer can be calculated and so forth until the output +is obtained.
@@ -390,8 +399,9 @@ $$
-We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form -is, +We can generalize this expression to an MLP with \( l \) hidden +layers. The complete functional form is, + $$ \begin{align} &y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\! @@ -402,7 +412,9 @@ $$ \end{align} $$ -which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \). +
+which illustrates a basic property of MLPs: The only independent +variables are the input values \( x_n \).
@@ -410,15 +422,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
-This confirms that an MLP, -despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a -mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \). -In our example, \( n=2 \) and \( m=1 \). Consequentially, -the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. +This confirms that an MLP, despite its quite convoluted mathematical +form, is nothing more than an analytic function, specifically a +mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow +\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and +\( m=1 \). Consequentially, the number of input and output values of the +function we want to fit must be equal to the number of inputs and +outputs of our MLP.
-Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that -the expression is essentially a nested sum of scaled activation functions of the form +Furthermore, the flexibility and universality of a MLP can be +illustrated by realizing that the expression is essentially a nested +sum of scaled activation functions of the form $$ \begin{equation} @@ -427,15 +442,18 @@ $$ \end{equation} $$ -where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions -can be shifted up and down or left and right, change slope or be rescaled -which is the key to the flexibility of a neural network. +
+where the parameters \( c_i \) are weights and biases. By adjusting these +parameters, the activation functions can be shifted up and down or +left and right, change slope or be rescaled which is the key to the +flexibility of a neural network.
We can introduce a more convenient notation for the activations in a NN.
@@ -474,6 +492,7 @@ $$
The activation of node \( i \) in layer 2 is $$ @@ -484,9 +503,11 @@ $$ \end{equation} $$ -This is not just a convenient and compact notation, but also -a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications -and vector additions that are used as input to the activation functions. For each operation +
+This is not just a convenient and compact notation, but also a useful +and intuitive way to think about MLPs: The output is calculated by a +series of matrix-vector multiplications and vector additions that are +used as input to the activation functions. For each operation \( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
@@ -495,9 +516,10 @@ and vector additions that are used as input to the activation functions. For eac
-A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). -As described in, the following restrictions are imposed on an activation function for a FFNN -to fulfill the universal approximation theorem +A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). As described +in, the following restrictions are imposed on an activation function +for a FFNN to fulfill the universal approximation theorem
-The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each -layer simply performs a linear transformation of its inputs. +The second requirement excludes all linear functions. Furthermore, in +a MLP with only linear activation functions, each layer simply +performs a linear transformation of its inputs.
-Regardless of the number of layers, -the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of -non-linearity to the NN to be able to fit non-linear functions -Typical examples are the logistic Sigmoid +Regardless of the number of layers, the output of the NN will be +nothing but a linear function of the inputs. Thus we need to introduce +some kind of non-linearity to the NN to be able to fit non-linear +functions Typical examples are the logistic Sigmoid + $$ \begin{equation} f(x) = \frac{1}{1 + e^{-x}}, @@ -539,11 +563,12 @@ $$
+The sigmoid function are more biologically plausible because the +output of inactive neurons are zero. Such activation function are +called one-sided. However, it has been shown that the hyperbolic +tangent performs better than the sigmoid for training MLPs. has +become the most popular for deep neural networks
diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html index b01022ab1..39ae1aa1e 100644 --- a/doc/pub/NeuralNet/html/NeuralNet.html +++ b/doc/pub/NeuralNet/html/NeuralNet.html @@ -269,19 +269,16 @@ means that each node in the output layer has a linear activation function). The there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as a separate type of NN due the unusual activation functions. -
-Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description -of how a fully-connected FFNN works, and how it can be used to interpolate data sets. -
-One use often so-called fully-connected feed-forward neural networks with three -or more layers (an input layer, one or more hidden layers and an output layer) -consisting of neurons that have non-linear activation functions. +One uses often so-called fully-connected feed-forward neural networks +with three or more layers (an input layer, one or more hidden layers +and an output layer) consisting of neurons that have non-linear +activation functions.
Such networks are often called multilayer perceptrons (MLPs) @@ -295,17 +292,22 @@ Such networks are often called multilayer perceptrons (MLPs) According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function. -Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always -assumed to be linear, so as to not restrict the range of output values.
-We note that this theorem is only applicable to a NN with one hidden layer. -Therefore, we can easily construct an NN -that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer -with activation functions that do. Furthermore, although the universal approximation theorem -lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: -A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other -functions. +Note that the requirements on the activation function only applies to +the hidden layer, the output nodes are always assumed to be linear, so +as to not restrict the range of output values. + +
+We note that this theorem is only applicable to an NN with one hidden +layer. Therefore, we can easily construct an NN that employs +activation functions which do not satisfy the above requirements, as +long as we have at least one layer with activation functions that +do. Furthermore, although the universal approximation theorem lays +the theoretical foundation for regression with neural networks, it +does not say anything about how things work in practice: A neural +network can still be able to approximate a given function reasonably +well without having the flexibility to fit all other functions.
@@ -319,9 +321,11 @@ $$
\end{equation}
$$
-In an FFNN of such neurons, the inputs \( x_i \)
-are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
-which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer.
+
+In an FFNN of such neurons, the inputs \( x_i \) are the outputs of +the neurons in the preceding layer. Furthermore, an MLP is +fully-connected, which means that each neuron receives a weighted sum +of the outputs of all neurons in the previous layer.
@@ -348,7 +352,9 @@ $$
\end{equation}
$$
-where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
+
+where we assume that all nodes in the same layer have identical +activation functions, hence the notation \( f_l \) $$ \begin{equation} @@ -357,8 +363,11 @@ $$ \end{equation} $$ -where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed, -the values of the subsequent layer can be calculated and so forth until the output is obtained. +
+where \( N_l \) is the number of nodes in layer \( l \). When the output of +all the nodes in the first hidden layer are computed, the values of +the subsequent layer can be calculated and so forth until the output +is obtained.
@@ -395,8 +404,9 @@ $$
-We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form -is, +We can generalize this expression to an MLP with \( l \) hidden +layers. The complete functional form is, + $$ \begin{align} &y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\! @@ -407,7 +417,9 @@ $$ \end{align} $$ -which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \). +
+which illustrates a basic property of MLPs: The only independent +variables are the input values \( x_n \).
@@ -415,15 +427,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
-This confirms that an MLP, -despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a -mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \). -In our example, \( n=2 \) and \( m=1 \). Consequentially, -the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. +This confirms that an MLP, despite its quite convoluted mathematical +form, is nothing more than an analytic function, specifically a +mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow +\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and +\( m=1 \). Consequentially, the number of input and output values of the +function we want to fit must be equal to the number of inputs and +outputs of our MLP.
-Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that -the expression is essentially a nested sum of scaled activation functions of the form +Furthermore, the flexibility and universality of a MLP can be +illustrated by realizing that the expression is essentially a nested +sum of scaled activation functions of the form $$ \begin{equation} @@ -432,15 +447,18 @@ $$ \end{equation} $$ -where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions -can be shifted up and down or left and right, change slope or be rescaled -which is the key to the flexibility of a neural network. +
+where the parameters \( c_i \) are weights and biases. By adjusting these +parameters, the activation functions can be shifted up and down or +left and right, change slope or be rescaled which is the key to the +flexibility of a neural network.
We can introduce a more convenient notation for the activations in a NN.
@@ -479,6 +497,7 @@ $$
The activation of node \( i \) in layer 2 is $$ @@ -489,9 +508,11 @@ $$ \end{equation} $$ -This is not just a convenient and compact notation, but also -a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications -and vector additions that are used as input to the activation functions. For each operation +
+This is not just a convenient and compact notation, but also a useful +and intuitive way to think about MLPs: The output is calculated by a +series of matrix-vector multiplications and vector additions that are +used as input to the activation functions. For each operation \( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
@@ -500,9 +521,10 @@ and vector additions that are used as input to the activation functions. For eac
-A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). -As described in, the following restrictions are imposed on an activation function for a FFNN -to fulfill the universal approximation theorem +A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). As described +in, the following restrictions are imposed on an activation function +for a FFNN to fulfill the universal approximation theorem
-The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each -layer simply performs a linear transformation of its inputs. +The second requirement excludes all linear functions. Furthermore, in +a MLP with only linear activation functions, each layer simply +performs a linear transformation of its inputs.
-Regardless of the number of layers, -the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of -non-linearity to the NN to be able to fit non-linear functions -Typical examples are the logistic Sigmoid +Regardless of the number of layers, the output of the NN will be +nothing but a linear function of the inputs. Thus we need to introduce +some kind of non-linearity to the NN to be able to fit non-linear +functions Typical examples are the logistic Sigmoid + $$ \begin{equation} f(x) = \frac{1}{1 + e^{-x}}, @@ -544,11 +568,12 @@ $$
+The sigmoid function are more biologically plausible because the +output of inactive neurons are zero. Such activation function are +called one-sided. However, it has been shown that the hyperbolic +tangent performs better than the sigmoid for training MLPs. has +become the most popular for deep neural networks
diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb index 5665fe4cd..da8ec380c 100644 --- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb +++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb @@ -161,15 +161,12 @@ "there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as\n", "a separate type of NN due the unusual activation functions.\n", "\n", - "\n", - "Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description\n", - "of how a fully-connected FFNN works, and how it can be used to interpolate data sets. \n", - "\n", "## Multilayer perceptrons\n", "\n", - "One use often so-called fully-connected feed-forward neural networks with three\n", - "or more layers (an input layer, one or more hidden layers and an output layer)\n", - "consisting of neurons that have non-linear activation functions.\n", + "One uses often so-called fully-connected feed-forward neural networks\n", + "with three or more layers (an input layer, one or more hidden layers\n", + "and an output layer) consisting of neurons that have non-linear\n", + "activation functions.\n", "\n", "Such networks are often called *multilayer perceptrons* (MLPs)\n", "\n", @@ -178,16 +175,20 @@ "According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing \n", "a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, \n", "assuming the activation function for the hidden layer is a **non-constant, bounded and monotonically-increasing continuous function**.\n", - "Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always\n", - "assumed to be linear, so as to not restrict the range of output values. \n", "\n", - "We note that this theorem is only applicable to a NN with *one* hidden layer. \n", - "Therefore, we can easily construct an NN \n", - "that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer\n", - "with activation functions that *do*. Furthermore, although the universal approximation theorem\n", - "lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: \n", - "A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit *all other*\n", - "functions. \n", + "Note that the requirements on the activation function only applies to\n", + "the hidden layer, the output nodes are always assumed to be linear, so\n", + "as to not restrict the range of output values.\n", + "\n", + "We note that this theorem is only applicable to an NN with *one* hidden\n", + "layer. Therefore, we can easily construct an NN that employs\n", + "activation functions which do not satisfy the above requirements, as\n", + "long as we have at least one layer with activation functions that\n", + "*do*. Furthermore, although the universal approximation theorem lays\n", + "the theoretical foundation for regression with neural networks, it\n", + "does not say anything about how things work in practice: A neural\n", + "network can still be able to approximate a given function reasonably\n", + "well without having the flexibility to fit *all other* functions.\n", "\n", "\n", "\n", @@ -213,9 +214,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In an FFNN of such neurons, the *inputs* $x_i$\n", - "are the *outputs* of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, \n", - "which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer. \n", + "In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of\n", + "the neurons in the preceding layer. Furthermore, an MLP is\n", + "fully-connected, which means that each neuron receives a weighted sum\n", + "of the outputs of *all* neurons in the previous layer.\n", "\n", "## Mathematical model\n", "\n", @@ -264,7 +266,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where we assume that all nodes in the same layer have identical activation functions, hence the notation $f_l$" + "where we assume that all nodes in the same layer have identical\n", + "activation functions, hence the notation $f_l$" ] }, { @@ -286,8 +289,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $N_l$ is the number of nodes in layer $l$. When the output of all the nodes in the first hidden layer are computed,\n", - "the values of the subsequent layer can be calculated and so forth until the output is obtained. \n", + "where $N_l$ is the number of nodes in layer $l$. When the output of\n", + "all the nodes in the first hidden layer are computed, the values of\n", + "the subsequent layer can be calculated and so forth until the output\n", + "is obtained.\n", "\n", "\n", "\n", @@ -370,8 +375,8 @@ "source": [ "## Mathematical model\n", "\n", - "We can generalize this expression to an MLP with $l$ hidden layers. The complete functional form\n", - "is," + "We can generalize this expression to an MLP with $l$ hidden\n", + "layers. The complete functional form is," ] }, { @@ -396,18 +401,22 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "which illustrates a basic property of MLPs: The only independent variables are the input values $x_n$. \n", + "which illustrates a basic property of MLPs: The only independent\n", + "variables are the input values $x_n$.\n", "\n", "## Mathematical model\n", "\n", - "This confirms that an MLP,\n", - "despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a \n", - "mapping of real-valued vectors $\\vec{x} \\in \\mathbb{R}^n \\rightarrow \\vec{y} \\in \\mathbb{R}^m$. \n", - "In our example, $n=2$ and $m=1$. Consequentially, \n", - "the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. \n", + "This confirms that an MLP, despite its quite convoluted mathematical\n", + "form, is nothing more than an analytic function, specifically a\n", + "mapping of real-valued vectors $\\vec{x} \\in \\mathbb{R}^n \\rightarrow\n", + "\\vec{y} \\in \\mathbb{R}^m$. In our example, $n=2$ and\n", + "$m=1$. Consequentially, the number of input and output values of the\n", + "function we want to fit must be equal to the number of inputs and\n", + "outputs of our MLP.\n", "\n", - "Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that \n", - "the expression is essentially a nested sum of scaled activation functions of the form" + "Furthermore, the flexibility and universality of a MLP can be\n", + "illustrated by realizing that the expression is essentially a nested\n", + "sum of scaled activation functions of the form" ] }, { @@ -429,9 +438,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions\n", - "can be shifted up and down or left and right, change slope or be rescaled \n", - "which is the key to the flexibility of a neural network. \n", + "where the parameters $c_i$ are weights and biases. By adjusting these\n", + "parameters, the activation functions can be shifted up and down or\n", + "left and right, change slope or be rescaled which is the key to the\n", + "flexibility of a neural network.\n", "\n", "### Matrix-vector notation\n", "\n", @@ -505,17 +515,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This is not just a convenient and compact notation, but also \n", - "a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications\n", - "and vector additions that are used as input to the activation functions. For each operation \n", - "$\\mathrm{W}_l \\vec{y}_{l-1}$ we move forward one layer. \n", + "This is not just a convenient and compact notation, but also a useful\n", + "and intuitive way to think about MLPs: The output is calculated by a\n", + "series of matrix-vector multiplications and vector additions that are\n", + "used as input to the activation functions. For each operation\n", + "$\\mathrm{W}_l \\vec{y}_{l-1}$ we move forward one layer.\n", "\n", "\n", "### Activation functions\n", "\n", - "A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). \n", - "As described in, the following restrictions are imposed on an activation function for a FFNN\n", - "to fulfill the universal approximation theorem\n", + "A property that characterizes a neural network, other than its\n", + "connectivity, is the choice of activation function(s). As described\n", + "in, the following restrictions are imposed on an activation function\n", + "for a FFNN to fulfill the universal approximation theorem\n", "\n", " * Non-constant\n", "\n", @@ -527,13 +539,14 @@ "\n", "### Activation functions, Logistic and Hyperbolic ones\n", "\n", - "The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each \n", - "layer simply performs a linear transformation of its inputs.\n", + "The second requirement excludes all linear functions. Furthermore, in\n", + "a MLP with only linear activation functions, each layer simply\n", + "performs a linear transformation of its inputs.\n", "\n", - "Regardless of the number of layers, \n", - "the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of \n", - "non-linearity to the NN to be able to fit non-linear functions\n", - "Typical examples are the logistic *Sigmoid*" + "Regardless of the number of layers, the output of the NN will be\n", + "nothing but a linear function of the inputs. Thus we need to introduce\n", + "some kind of non-linearity to the NN to be able to fit non-linear\n", + "functions Typical examples are the logistic *Sigmoid*" ] }, { @@ -579,11 +592,11 @@ "source": [ "### Relevance\n", "\n", - "The *sigmoid* function are more biologically plausible because \n", - "the output of inactive neurons are zero. Such activation function are called *one-sided*. However,\n", - "it has been shown that the hyperbolic tangent \n", - "performs better than the sigmoid for training MLPs. \n", - "has become the most popular for *deep neural networks*" + "The *sigmoid* function are more biologically plausible because the\n", + "output of inactive neurons are zero. Such activation function are\n", + "called *one-sided*. However, it has been shown that the hyperbolic\n", + "tangent performs better than the sigmoid for training MLPs. has\n", + "become the most popular for *deep neural networks*" ] }, { diff --git a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz index b35e004cf..7339c4e91 100644 Binary files a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz and b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz differ diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-beamer-handouts2x3.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-beamer-handouts2x3.pdf index 646de64e9..26c2e2b67 100644 Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-beamer-handouts2x3.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-beamer-handouts2x3.pdf differ diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-beamer.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-beamer.pdf index 613603e9a..ad2a32bac 100644 Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-beamer.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-beamer.pdf differ diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf index 24a8b5459..15f482553 100644 Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf differ diff --git a/doc/src/NeuralNet/NeuralNet.do.txt b/doc/src/NeuralNet/NeuralNet.do.txt index 99bd95258..c3ac45f5d 100644 --- a/doc/src/NeuralNet/NeuralNet.do.txt +++ b/doc/src/NeuralNet/NeuralNet.do.txt @@ -137,16 +137,13 @@ means that each node in the output layer has a linear activation function). The there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as a separate type of NN due the unusual activation functions. - -Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description -of how a fully-connected FFNN works, and how it can be used to interpolate data sets. - !split ===== Multilayer perceptrons ===== -One use often so-called fully-connected feed-forward neural networks with three -or more layers (an input layer, one or more hidden layers and an output layer) -consisting of neurons that have non-linear activation functions. +One uses often so-called fully-connected feed-forward neural networks +with three or more layers (an input layer, one or more hidden layers +and an output layer) consisting of neurons that have non-linear +activation functions. Such networks are often called *multilayer perceptrons* (MLPs) @@ -156,16 +153,20 @@ Such networks are often called *multilayer perceptrons* (MLPs) According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, assuming the activation function for the hidden layer is a _non-constant, bounded and monotonically-increasing continuous function_. -Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always -assumed to be linear, so as to not restrict the range of output values. -We note that this theorem is only applicable to a NN with *one* hidden layer. -Therefore, we can easily construct an NN -that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer -with activation functions that *do*. Furthermore, although the universal approximation theorem -lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: -A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit *all other* -functions. +Note that the requirements on the activation function only applies to +the hidden layer, the output nodes are always assumed to be linear, so +as to not restrict the range of output values. + +We note that this theorem is only applicable to an NN with *one* hidden +layer. Therefore, we can easily construct an NN that employs +activation functions which do not satisfy the above requirements, as +long as we have at least one layer with activation functions that +*do*. Furthermore, although the universal approximation theorem lays +the theoretical foundation for regression with neural networks, it +does not say anything about how things work in practice: A neural +network can still be able to approximate a given function reasonably +well without having the flexibility to fit *all other* functions. @@ -178,9 +179,11 @@ functions. label{artificialNeuron2} \end{equation} !et -In an FFNN of such neurons, the *inputs* $x_i$ -are the *outputs* of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, -which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer. + +In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of +the neurons in the preceding layer. Furthermore, an MLP is +fully-connected, which means that each neuron receives a weighted sum +of the outputs of *all* neurons in the previous layer. !split ===== Mathematical model ===== @@ -201,7 +204,9 @@ producing the output $y_i^1$ of all neurons in layer 1, label{outputLayer1} \end{equation} !et -where we assume that all nodes in the same layer have identical activation functions, hence the notation $f_l$ + +where we assume that all nodes in the same layer have identical +activation functions, hence the notation $f_l$ !bt \begin{equation} @@ -209,8 +214,11 @@ where we assume that all nodes in the same layer have identical activation funct label{generalLayer} \end{equation} !et -where $N_l$ is the number of nodes in layer $l$. When the output of all the nodes in the first hidden layer are computed, -the values of the subsequent layer can be calculated and so forth until the output is obtained. + +where $N_l$ is the number of nodes in layer $l$. When the output of +all the nodes in the first hidden layer are computed, the values of +the subsequent layer can be calculated and so forth until the output +is obtained. @@ -239,8 +247,9 @@ where we have substituted $y_m^1$ with. Finally, the NN output yields, !split ===== Mathematical model ===== -We can generalize this expression to an MLP with $l$ hidden layers. The complete functional form -is, +We can generalize this expression to an MLP with $l$ hidden +layers. The complete functional form is, + !bt \begin{align} &y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\! @@ -250,31 +259,39 @@ is, label{completeNN} \end{align} !et -which illustrates a basic property of MLPs: The only independent variables are the input values $x_n$. + +which illustrates a basic property of MLPs: The only independent +variables are the input values $x_n$. !split ===== Mathematical model ===== -This confirms that an MLP, -despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a -mapping of real-valued vectors $\vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m$. -In our example, $n=2$ and $m=1$. Consequentially, -the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. +This confirms that an MLP, despite its quite convoluted mathematical +form, is nothing more than an analytic function, specifically a +mapping of real-valued vectors $\vec{x} \in \mathbb{R}^n \rightarrow +\vec{y} \in \mathbb{R}^m$. In our example, $n=2$ and +$m=1$. Consequentially, the number of input and output values of the +function we want to fit must be equal to the number of inputs and +outputs of our MLP. -Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that -the expression is essentially a nested sum of scaled activation functions of the form +Furthermore, the flexibility and universality of a MLP can be +illustrated by realizing that the expression is essentially a nested +sum of scaled activation functions of the form !bt \begin{equation} h(x) = c_1 f(c_2 x + c_3) + c_4 \end{equation} !et -where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions -can be shifted up and down or left and right, change slope or be rescaled -which is the key to the flexibility of a neural network. + +where the parameters $c_i$ are weights and biases. By adjusting these +parameters, the activation functions can be shifted up and down or +left and right, change slope or be rescaled which is the key to the +flexibility of a neural network. !split === Matrix-vector notation === + We can introduce a more convenient notation for the activations in a NN. Additionally, we can represent the biases and activations @@ -307,6 +324,7 @@ the equation for the activations of hidden layer 2 in !split === Matrix-vector notation and activation === + The activation of node $i$ in layer 2 is !bt @@ -315,19 +333,22 @@ The activation of node $i$ in layer 2 is f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right). \end{equation} !et -This is not just a convenient and compact notation, but also -a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications -and vector additions that are used as input to the activation functions. For each operation -$\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer. + +This is not just a convenient and compact notation, but also a useful +and intuitive way to think about MLPs: The output is calculated by a +series of matrix-vector multiplications and vector additions that are +used as input to the activation functions. For each operation +$\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer. !split === Activation functions === -A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). -As described in, the following restrictions are imposed on an activation function for a FFNN -to fulfill the universal approximation theorem +A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). As described +in, the following restrictions are imposed on an activation function +for a FFNN to fulfill the universal approximation theorem * Non-constant @@ -340,13 +361,15 @@ to fulfill the universal approximation theorem !split === Activation functions, Logistic and Hyperbolic ones === -The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each -layer simply performs a linear transformation of its inputs. +The second requirement excludes all linear functions. Furthermore, in +a MLP with only linear activation functions, each layer simply +performs a linear transformation of its inputs. + +Regardless of the number of layers, the output of the NN will be +nothing but a linear function of the inputs. Thus we need to introduce +some kind of non-linearity to the NN to be able to fit non-linear +functions Typical examples are the logistic *Sigmoid* -Regardless of the number of layers, -the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of -non-linearity to the NN to be able to fit non-linear functions -Typical examples are the logistic *Sigmoid* !bt \begin{equation} f(x) = \frac{1}{1 + e^{-x}}, @@ -363,11 +386,12 @@ and the *hyperbolic tangent* function !split === Relevance === -The *sigmoid* function are more biologically plausible because -the output of inactive neurons are zero. Such activation function are called *one-sided*. However, -it has been shown that the hyperbolic tangent -performs better than the sigmoid for training MLPs. -has become the most popular for *deep neural networks* + +The *sigmoid* function are more biologically plausible because the +output of inactive neurons are zero. Such activation function are +called *one-sided*. However, it has been shown that the hyperbolic +tangent performs better than the sigmoid for training MLPs. has +become the most popular for *deep neural networks* !bc pycod """The sigmoid function (or the logistic curve) is a