lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleaning up

Browse Source
This commit is contained in:
mhjensen
2018-09-28 11:24:22 +02:00
parent 4f225ce503
commit b59bb722d3
10 changed files with 451 additions and 314 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/NeuralNet-bs.html
+77 -52
View File
@@ -322,19 +322,16 @@ means that each node in the output layer has a linear activation function). The
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
a separate type of NN due the unusual activation functions.
<p>
Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
of how a fully-connected FFNN works, and how it can be used to interpolate data sets.
<p>
<!-- !split -->
<h2 id="___sec6" class="anchor">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
or more layers (an input layer, one or more hidden layers and an output layer)
consisting of neurons that have non-linear activation functions.
One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.
<p>
Such networks are often called <em>multilayer perceptrons</em> (MLPs)
@@ -348,17 +345,22 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
assuming the activation function for the hidden layer is a <b>non-constant, bounded and monotonically-increasing continuous function</b>.
Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
assumed to be linear, so as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to a NN with <em>one</em> hidden layer.
Therefore, we can easily construct an NN
that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
with activation functions that <em>do</em>. Furthermore, although the universal approximation theorem
lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit <em>all other</em>
functions.
Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to an NN with <em>one</em> hidden
layer. Therefore, we can easily construct an NN that employs
activation functions which do not satisfy the above requirements, as
long as we have at least one layer with activation functions that
<em>do</em>. Furthermore, although the universal approximation theorem lays
the theoretical foundation for regression with neural networks, it
does not say anything about how things work in practice: A neural
network can still be able to approximate a given function reasonably
well without having the flexibility to fit <em>all other</em> functions.
<p>
<!-- !split -->
@@ -372,9 +374,11 @@ $$
\end{equation}
$$
In an FFNN of such neurons, the <em>inputs</em> \( x_i \)
are the <em>outputs</em> of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of <em>all</em> neurons in the previous layer.
<p>
In an FFNN of such neurons, the <em>inputs</em> \( x_i \) are the <em>outputs</em> of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
of the outputs of <em>all</em> neurons in the previous layer.
<p>
<!-- !split -->
@@ -401,7 +405,9 @@ $$
\end{equation}
$$
where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
<p>
where we assume that all nodes in the same layer have identical
activation functions, hence the notation \( f_l \)
$$
\begin{equation}
@@ -410,8 +416,11 @@ $$
\end{equation}
$$
where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed,
the values of the subsequent layer can be calculated and so forth until the output is obtained.
<p>
where \( N_l \) is the number of nodes in layer \( l \). When the output of
all the nodes in the first hidden layer are computed, the values of
the subsequent layer can be calculated and so forth until the output
is obtained.
<p>
<!-- !split -->
@@ -448,8 +457,9 @@ $$
<h2 id="___sec11" class="anchor">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
is,
We can generalize this expression to an MLP with \( l \) hidden
layers. The complete functional form is,
$$
\begin{align}
&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\!
@@ -460,7 +470,9 @@ $$
\end{align}
$$
which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \).
<p>
which illustrates a basic property of MLPs: The only independent
variables are the input values \( x_n \).
<p>
<!-- !split -->
@@ -468,15 +480,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
<h2 id="___sec12" class="anchor">Mathematical model </h2>
<p>
This confirms that an MLP,
despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \).
In our example, \( n=2 \) and \( m=1 \). Consequentially,
the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP.
This confirms that an MLP, despite its quite convoluted mathematical
form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow
\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and
\( m=1 \). Consequentially, the number of input and output values of the
function we want to fit must be equal to the number of inputs and
outputs of our MLP.
<p>
Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that
the expression is essentially a nested sum of scaled activation functions of the form
Furthermore, the flexibility and universality of a MLP can be
illustrated by realizing that the expression is essentially a nested
sum of scaled activation functions of the form
$$
\begin{equation}
@@ -485,15 +500,18 @@ $$
\end{equation}
$$
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
which is the key to the flexibility of a neural network.
<p>
where the parameters \( c_i \) are weights and biases. By adjusting these
parameters, the activation functions can be shifted up and down or
left and right, change slope or be rescaled which is the key to the
flexibility of a neural network.
<p>
<!-- !split -->
<h3 id="___sec13" class="anchor">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in a NN.
<p>
@@ -532,6 +550,7 @@ $$
<h3 id="___sec14" class="anchor">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
$$
@@ -542,9 +561,11 @@ $$
\end{equation}
$$
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
<p>
This is not just a convenient and compact notation, but also a useful
and intuitive way to think about MLPs: The output is calculated by a
series of matrix-vector multiplications and vector additions that are
used as input to the activation functions. For each operation
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
<p>
@@ -553,9 +574,10 @@ and vector additions that are used as input to the activation functions. For eac
<h3 id="___sec15" class="anchor">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
As described in, the following restrictions are imposed on an activation function for a FFNN
to fulfill the universal approximation theorem
A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
<ul>
<li> Non-constant</li>
@@ -569,14 +591,16 @@ to fulfill the universal approximation theorem
<h3 id="___sec16" class="anchor">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
layer simply performs a linear transformation of its inputs.
The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.
<p>
Regardless of the number of layers,
the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
non-linearity to the NN to be able to fit non-linear functions
Typical examples are the logistic <em>Sigmoid</em>
Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic <em>Sigmoid</em>
$$
\begin{equation}
f(x) = \frac{1}{1 + e^{-x}},
@@ -597,11 +621,12 @@ $$
<h3 id="___sec17" class="anchor">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
it has been shown that the hyperbolic tangent
performs better than the sigmoid for training MLPs.
has become the most popular for <em>deep neural networks</em>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called <em>one-sided</em>. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs. has
become the most popular for <em>deep neural networks</em>
<p>
doc/pub/NeuralNet/html/NeuralNet-reveal.html
+77 -52
View File
@@ -319,10 +319,6 @@ an input layer, a hidden layer with non-linear radial symmetric activation funct
means that each node in the output layer has a linear activation function). The layers are normally fully-connected and
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
a separate type of NN due the unusual activation functions.
<p>
Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
of how a fully-connected FFNN works, and how it can be used to interpolate data sets.
</section>
@@ -330,9 +326,10 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
or more layers (an input layer, one or more hidden layers and an output layer)
consisting of neurons that have non-linear activation functions.
One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.
<p>
Such networks are often called <em>multilayer perceptrons</em> (MLPs)
@@ -346,17 +343,22 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
assuming the activation function for the hidden layer is a <b>non-constant, bounded and monotonically-increasing continuous function</b>.
Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
assumed to be linear, so as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to a NN with <em>one</em> hidden layer.
Therefore, we can easily construct an NN
that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
with activation functions that <em>do</em>. Furthermore, although the universal approximation theorem
lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit <em>all other</em>
functions.
Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to an NN with <em>one</em> hidden
layer. Therefore, we can easily construct an NN that employs
activation functions which do not satisfy the above requirements, as
long as we have at least one layer with activation functions that
<em>do</em>. Furthermore, although the universal approximation theorem lays
the theoretical foundation for regression with neural networks, it
does not say anything about how things work in practice: A neural
network can still be able to approximate a given function reasonably
well without having the flexibility to fit <em>all other</em> functions.
</section>
@@ -372,9 +374,11 @@ $$
$$
<p>&nbsp;<br>
In an FFNN of such neurons, the <em>inputs</em> \( x_i \)
are the <em>outputs</em> of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of <em>all</em> neurons in the previous layer.
<p>
In an FFNN of such neurons, the <em>inputs</em> \( x_i \) are the <em>outputs</em> of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
of the outputs of <em>all</em> neurons in the previous layer.
</section>
@@ -405,7 +409,9 @@ $$
$$
<p>&nbsp;<br>
where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
<p>
where we assume that all nodes in the same layer have identical
activation functions, hence the notation \( f_l \)
<p>&nbsp;<br>
$$
@@ -416,8 +422,11 @@ $$
$$
<p>&nbsp;<br>
where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed,
the values of the subsequent layer can be calculated and so forth until the output is obtained.
<p>
where \( N_l \) is the number of nodes in layer \( l \). When the output of
all the nodes in the first hidden layer are computed, the values of
the subsequent layer can be calculated and so forth until the output
is obtained.
</section>
@@ -458,8 +467,9 @@ $$
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
is,
We can generalize this expression to an MLP with \( l \) hidden
layers. The complete functional form is,
<p>&nbsp;<br>
$$
\begin{align}
@@ -472,7 +482,9 @@ $$
$$
<p>&nbsp;<br>
which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \).
<p>
which illustrates a basic property of MLPs: The only independent
variables are the input values \( x_n \).
</section>
@@ -480,15 +492,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP,
despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \).
In our example, \( n=2 \) and \( m=1 \). Consequentially,
the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP.
This confirms that an MLP, despite its quite convoluted mathematical
form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow
\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and
\( m=1 \). Consequentially, the number of input and output values of the
function we want to fit must be equal to the number of inputs and
outputs of our MLP.
<p>
Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that
the expression is essentially a nested sum of scaled activation functions of the form
Furthermore, the flexibility and universality of a MLP can be
illustrated by realizing that the expression is essentially a nested
sum of scaled activation functions of the form
<p>&nbsp;<br>
$$
@@ -499,9 +514,11 @@ $$
$$
<p>&nbsp;<br>
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
which is the key to the flexibility of a neural network.
<p>
where the parameters \( c_i \) are weights and biases. By adjusting these
parameters, the activation functions can be shifted up and down or
left and right, change slope or be rescaled which is the key to the
flexibility of a neural network.
</section>
@@ -509,6 +526,7 @@ which is the key to the flexibility of a neural network.
<h3 id="___sec13">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in a NN.
<p>
@@ -550,6 +568,7 @@ $$
<h3 id="___sec14">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
<p>&nbsp;<br>
@@ -562,9 +581,11 @@ $$
$$
<p>&nbsp;<br>
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
<p>
This is not just a convenient and compact notation, but also a useful
and intuitive way to think about MLPs: The output is calculated by a
series of matrix-vector multiplications and vector additions that are
used as input to the activation functions. For each operation
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
</section>
@@ -574,9 +595,10 @@ and vector additions that are used as input to the activation functions. For eac
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
As described in, the following restrictions are imposed on an activation function for a FFNN
to fulfill the universal approximation theorem
A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
<ul>
@@ -596,14 +618,16 @@ to fulfill the universal approximation theorem
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
layer simply performs a linear transformation of its inputs.
The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.
<p>
Regardless of the number of layers,
the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
non-linearity to the NN to be able to fit non-linear functions
Typical examples are the logistic <em>Sigmoid</em>
Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic <em>Sigmoid</em>
<p>&nbsp;<br>
$$
\begin{equation}
@@ -629,11 +653,12 @@ $$
<h3 id="___sec17">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
it has been shown that the hyperbolic tangent
performs better than the sigmoid for training MLPs.
has become the most popular for <em>deep neural networks</em>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called <em>one-sided</em>. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs. has
become the most popular for <em>deep neural networks</em>
<p>
doc/pub/NeuralNet/html/NeuralNet-solarized.html
+77 -52
View File
@@ -264,19 +264,16 @@ means that each node in the output layer has a linear activation function). The
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
a separate type of NN due the unusual activation functions.
<p>
Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
of how a fully-connected FFNN works, and how it can be used to interpolate data sets.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
or more layers (an input layer, one or more hidden layers and an output layer)
consisting of neurons that have non-linear activation functions.
One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.
<p>
Such networks are often called <em>multilayer perceptrons</em> (MLPs)
@@ -290,17 +287,22 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
assuming the activation function for the hidden layer is a <b>non-constant, bounded and monotonically-increasing continuous function</b>.
Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
assumed to be linear, so as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to a NN with <em>one</em> hidden layer.
Therefore, we can easily construct an NN
that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
with activation functions that <em>do</em>. Furthermore, although the universal approximation theorem
lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit <em>all other</em>
functions.
Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to an NN with <em>one</em> hidden
layer. Therefore, we can easily construct an NN that employs
activation functions which do not satisfy the above requirements, as
long as we have at least one layer with activation functions that
<em>do</em>. Furthermore, although the universal approximation theorem lays
the theoretical foundation for regression with neural networks, it
does not say anything about how things work in practice: A neural
network can still be able to approximate a given function reasonably
well without having the flexibility to fit <em>all other</em> functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -314,9 +316,11 @@ $$
\end{equation}
$$
In an FFNN of such neurons, the <em>inputs</em> \( x_i \)
are the <em>outputs</em> of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of <em>all</em> neurons in the previous layer.
<p>
In an FFNN of such neurons, the <em>inputs</em> \( x_i \) are the <em>outputs</em> of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
of the outputs of <em>all</em> neurons in the previous layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -343,7 +347,9 @@ $$
\end{equation}
$$
where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
<p>
where we assume that all nodes in the same layer have identical
activation functions, hence the notation \( f_l \)
$$
\begin{equation}
@@ -352,8 +358,11 @@ $$
\end{equation}
$$
where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed,
the values of the subsequent layer can be calculated and so forth until the output is obtained.
<p>
where \( N_l \) is the number of nodes in layer \( l \). When the output of
all the nodes in the first hidden layer are computed, the values of
the subsequent layer can be calculated and so forth until the output
is obtained.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -390,8 +399,9 @@ $$
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
is,
We can generalize this expression to an MLP with \( l \) hidden
layers. The complete functional form is,
$$
\begin{align}
&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\!
@@ -402,7 +412,9 @@ $$
\end{align}
$$
which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \).
<p>
which illustrates a basic property of MLPs: The only independent
variables are the input values \( x_n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -410,15 +422,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP,
despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \).
In our example, \( n=2 \) and \( m=1 \). Consequentially,
the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP.
This confirms that an MLP, despite its quite convoluted mathematical
form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow
\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and
\( m=1 \). Consequentially, the number of input and output values of the
function we want to fit must be equal to the number of inputs and
outputs of our MLP.
<p>
Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that
the expression is essentially a nested sum of scaled activation functions of the form
Furthermore, the flexibility and universality of a MLP can be
illustrated by realizing that the expression is essentially a nested
sum of scaled activation functions of the form
$$
\begin{equation}
@@ -427,15 +442,18 @@ $$
\end{equation}
$$
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
which is the key to the flexibility of a neural network.
<p>
where the parameters \( c_i \) are weights and biases. By adjusting these
parameters, the activation functions can be shifted up and down or
left and right, change slope or be rescaled which is the key to the
flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec13">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in a NN.
<p>
@@ -474,6 +492,7 @@ $$
<h3 id="___sec14">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
$$
@@ -484,9 +503,11 @@ $$
\end{equation}
$$
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
<p>
This is not just a convenient and compact notation, but also a useful
and intuitive way to think about MLPs: The output is calculated by a
series of matrix-vector multiplications and vector additions that are
used as input to the activation functions. For each operation
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
<p>
@@ -495,9 +516,10 @@ and vector additions that are used as input to the activation functions. For eac
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
As described in, the following restrictions are imposed on an activation function for a FFNN
to fulfill the universal approximation theorem
A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
<ul>
<li> Non-constant</li>
@@ -511,14 +533,16 @@ to fulfill the universal approximation theorem
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
layer simply performs a linear transformation of its inputs.
The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.
<p>
Regardless of the number of layers,
the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
non-linearity to the NN to be able to fit non-linear functions
Typical examples are the logistic <em>Sigmoid</em>
Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic <em>Sigmoid</em>
$$
\begin{equation}
f(x) = \frac{1}{1 + e^{-x}},
@@ -539,11 +563,12 @@ $$
<h3 id="___sec17">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
it has been shown that the hyperbolic tangent
performs better than the sigmoid for training MLPs.
has become the most popular for <em>deep neural networks</em>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called <em>one-sided</em>. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs. has
become the most popular for <em>deep neural networks</em>
<p>
doc/pub/NeuralNet/html/NeuralNet.html
+77 -52
View File
@@ -269,19 +269,16 @@ means that each node in the output layer has a linear activation function). The
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
a separate type of NN due the unusual activation functions.
<p>
Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
of how a fully-connected FFNN works, and how it can be used to interpolate data sets.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
or more layers (an input layer, one or more hidden layers and an output layer)
consisting of neurons that have non-linear activation functions.
One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.
<p>
Such networks are often called <em>multilayer perceptrons</em> (MLPs)
@@ -295,17 +292,22 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
assuming the activation function for the hidden layer is a <b>non-constant, bounded and monotonically-increasing continuous function</b>.
Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
assumed to be linear, so as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to a NN with <em>one</em> hidden layer.
Therefore, we can easily construct an NN
that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
with activation functions that <em>do</em>. Furthermore, although the universal approximation theorem
lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit <em>all other</em>
functions.
Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to an NN with <em>one</em> hidden
layer. Therefore, we can easily construct an NN that employs
activation functions which do not satisfy the above requirements, as
long as we have at least one layer with activation functions that
<em>do</em>. Furthermore, although the universal approximation theorem lays
the theoretical foundation for regression with neural networks, it
does not say anything about how things work in practice: A neural
network can still be able to approximate a given function reasonably
well without having the flexibility to fit <em>all other</em> functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -319,9 +321,11 @@ $$
\end{equation}
$$
In an FFNN of such neurons, the <em>inputs</em> \( x_i \)
are the <em>outputs</em> of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of <em>all</em> neurons in the previous layer.
<p>
In an FFNN of such neurons, the <em>inputs</em> \( x_i \) are the <em>outputs</em> of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
of the outputs of <em>all</em> neurons in the previous layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -348,7 +352,9 @@ $$
\end{equation}
$$
where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
<p>
where we assume that all nodes in the same layer have identical
activation functions, hence the notation \( f_l \)
$$
\begin{equation}
@@ -357,8 +363,11 @@ $$
\end{equation}
$$
where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed,
the values of the subsequent layer can be calculated and so forth until the output is obtained.
<p>
where \( N_l \) is the number of nodes in layer \( l \). When the output of
all the nodes in the first hidden layer are computed, the values of
the subsequent layer can be calculated and so forth until the output
is obtained.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -395,8 +404,9 @@ $$
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
is,
We can generalize this expression to an MLP with \( l \) hidden
layers. The complete functional form is,
$$
\begin{align}
&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\!
@@ -407,7 +417,9 @@ $$
\end{align}
$$
which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \).
<p>
which illustrates a basic property of MLPs: The only independent
variables are the input values \( x_n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -415,15 +427,18 @@ which illustrates a basic property of MLPs: The only independent variables are t
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP,
despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \).
In our example, \( n=2 \) and \( m=1 \). Consequentially,
the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP.
This confirms that an MLP, despite its quite convoluted mathematical
form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow
\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and
\( m=1 \). Consequentially, the number of input and output values of the
function we want to fit must be equal to the number of inputs and
outputs of our MLP.
<p>
Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that
the expression is essentially a nested sum of scaled activation functions of the form
Furthermore, the flexibility and universality of a MLP can be
illustrated by realizing that the expression is essentially a nested
sum of scaled activation functions of the form
$$
\begin{equation}
@@ -432,15 +447,18 @@ $$
\end{equation}
$$
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
which is the key to the flexibility of a neural network.
<p>
where the parameters \( c_i \) are weights and biases. By adjusting these
parameters, the activation functions can be shifted up and down or
left and right, change slope or be rescaled which is the key to the
flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec13">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in a NN.
<p>
@@ -479,6 +497,7 @@ $$
<h3 id="___sec14">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
$$
@@ -489,9 +508,11 @@ $$
\end{equation}
$$
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
<p>
This is not just a convenient and compact notation, but also a useful
and intuitive way to think about MLPs: The output is calculated by a
series of matrix-vector multiplications and vector additions that are
used as input to the activation functions. For each operation
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
<p>
@@ -500,9 +521,10 @@ and vector additions that are used as input to the activation functions. For eac
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
As described in, the following restrictions are imposed on an activation function for a FFNN
to fulfill the universal approximation theorem
A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
<ul>
<li> Non-constant</li>
@@ -516,14 +538,16 @@ to fulfill the universal approximation theorem
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
layer simply performs a linear transformation of its inputs.
The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.
<p>
Regardless of the number of layers,
the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
non-linearity to the NN to be able to fit non-linear functions
Typical examples are the logistic <em>Sigmoid</em>
Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic <em>Sigmoid</em>
$$
\begin{equation}
f(x) = \frac{1}{1 + e^{-x}},
@@ -544,11 +568,12 @@ $$
<h3 id="___sec17">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
it has been shown that the hyperbolic tangent
performs better than the sigmoid for training MLPs.
has become the most popular for <em>deep neural networks</em>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called <em>one-sided</em>. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs. has
become the most popular for <em>deep neural networks</em>
<p>
doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
+66 -53
View File
@@ -161,15 +161,12 @@
"there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as\n",
"a separate type of NN due the unusual activation functions.\n",
"\n",
"\n",
"Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description\n",
"of how a fully-connected FFNN works, and how it can be used to interpolate data sets. \n",
"\n",
"## Multilayer perceptrons\n",
"\n",
"One use often so-called fully-connected feed-forward neural networks with three\n",
"or more layers (an input layer, one or more hidden layers and an output layer)\n",
"consisting of neurons that have non-linear activation functions.\n",
"One uses often so-called fully-connected feed-forward neural networks\n",
"with three or more layers (an input layer, one or more hidden layers\n",
"and an output layer) consisting of neurons that have non-linear\n",
"activation functions.\n",
"\n",
"Such networks are often called *multilayer perceptrons* (MLPs)\n",
"\n",
@@ -178,16 +175,20 @@
"According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing \n",
"a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, \n",
"assuming the activation function for the hidden layer is a **non-constant, bounded and monotonically-increasing continuous function**.\n",
"Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always\n",
"assumed to be linear, so as to not restrict the range of output values. \n",
"\n",
"We note that this theorem is only applicable to a NN with *one* hidden layer. \n",
"Therefore, we can easily construct an NN \n",
"that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer\n",
"with activation functions that *do*. Furthermore, although the universal approximation theorem\n",
"lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: \n",
"A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit *all other*\n",
"functions. \n",
"Note that the requirements on the activation function only applies to\n",
"the hidden layer, the output nodes are always assumed to be linear, so\n",
"as to not restrict the range of output values.\n",
"\n",
"We note that this theorem is only applicable to an NN with *one* hidden\n",
"layer. Therefore, we can easily construct an NN that employs\n",
"activation functions which do not satisfy the above requirements, as\n",
"long as we have at least one layer with activation functions that\n",
"*do*. Furthermore, although the universal approximation theorem lays\n",
"the theoretical foundation for regression with neural networks, it\n",
"does not say anything about how things work in practice: A neural\n",
"network can still be able to approximate a given function reasonably\n",
"well without having the flexibility to fit *all other* functions.\n",
"\n",
"\n",
"\n",
@@ -213,9 +214,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"In an FFNN of such neurons, the *inputs* $x_i$\n",
"are the *outputs* of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, \n",
"which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer. \n",
"In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of\n",
"the neurons in the preceding layer. Furthermore, an MLP is\n",
"fully-connected, which means that each neuron receives a weighted sum\n",
"of the outputs of *all* neurons in the previous layer.\n",
"\n",
"## Mathematical model\n",
"\n",
@@ -264,7 +266,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where we assume that all nodes in the same layer have identical activation functions, hence the notation $f_l$"
"where we assume that all nodes in the same layer have identical\n",
"activation functions, hence the notation $f_l$"
]
},
{
@@ -286,8 +289,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where $N_l$ is the number of nodes in layer $l$. When the output of all the nodes in the first hidden layer are computed,\n",
"the values of the subsequent layer can be calculated and so forth until the output is obtained. \n",
"where $N_l$ is the number of nodes in layer $l$. When the output of\n",
"all the nodes in the first hidden layer are computed, the values of\n",
"the subsequent layer can be calculated and so forth until the output\n",
"is obtained.\n",
"\n",
"\n",
"\n",
@@ -370,8 +375,8 @@
"source": [
"## Mathematical model\n",
"\n",
"We can generalize this expression to an MLP with $l$ hidden layers. The complete functional form\n",
"is,"
"We can generalize this expression to an MLP with $l$ hidden\n",
"layers. The complete functional form is,"
]
},
{
@@ -396,18 +401,22 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"which illustrates a basic property of MLPs: The only independent variables are the input values $x_n$. \n",
"which illustrates a basic property of MLPs: The only independent\n",
"variables are the input values $x_n$.\n",
"\n",
"## Mathematical model\n",
"\n",
"This confirms that an MLP,\n",
"despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a \n",
"mapping of real-valued vectors $\\vec{x} \\in \\mathbb{R}^n \\rightarrow \\vec{y} \\in \\mathbb{R}^m$. \n",
"In our example, $n=2$ and $m=1$. Consequentially, \n",
"the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. \n",
"This confirms that an MLP, despite its quite convoluted mathematical\n",
"form, is nothing more than an analytic function, specifically a\n",
"mapping of real-valued vectors $\\vec{x} \\in \\mathbb{R}^n \\rightarrow\n",
"\\vec{y} \\in \\mathbb{R}^m$. In our example, $n=2$ and\n",
"$m=1$. Consequentially, the number of input and output values of the\n",
"function we want to fit must be equal to the number of inputs and\n",
"outputs of our MLP.\n",
"\n",
"Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that \n",
"the expression is essentially a nested sum of scaled activation functions of the form"
"Furthermore, the flexibility and universality of a MLP can be\n",
"illustrated by realizing that the expression is essentially a nested\n",
"sum of scaled activation functions of the form"
]
},
{
@@ -429,9 +438,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions\n",
"can be shifted up and down or left and right, change slope or be rescaled \n",
"which is the key to the flexibility of a neural network. \n",
"where the parameters $c_i$ are weights and biases. By adjusting these\n",
"parameters, the activation functions can be shifted up and down or\n",
"left and right, change slope or be rescaled which is the key to the\n",
"flexibility of a neural network.\n",
"\n",
"### Matrix-vector notation\n",
"\n",
@@ -505,17 +515,19 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This is not just a convenient and compact notation, but also \n",
"a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications\n",
"and vector additions that are used as input to the activation functions. For each operation \n",
"$\\mathrm{W}_l \\vec{y}_{l-1}$ we move forward one layer. \n",
"This is not just a convenient and compact notation, but also a useful\n",
"and intuitive way to think about MLPs: The output is calculated by a\n",
"series of matrix-vector multiplications and vector additions that are\n",
"used as input to the activation functions. For each operation\n",
"$\\mathrm{W}_l \\vec{y}_{l-1}$ we move forward one layer.\n",
"\n",
"\n",
"### Activation functions\n",
"\n",
"A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). \n",
"As described in, the following restrictions are imposed on an activation function for a FFNN\n",
"to fulfill the universal approximation theorem\n",
"A property that characterizes a neural network, other than its\n",
"connectivity, is the choice of activation function(s). As described\n",
"in, the following restrictions are imposed on an activation function\n",
"for a FFNN to fulfill the universal approximation theorem\n",
"\n",
" * Non-constant\n",
"\n",
@@ -527,13 +539,14 @@
"\n",
"### Activation functions, Logistic and Hyperbolic ones\n",
"\n",
"The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each \n",
"layer simply performs a linear transformation of its inputs.\n",
"The second requirement excludes all linear functions. Furthermore, in\n",
"a MLP with only linear activation functions, each layer simply\n",
"performs a linear transformation of its inputs.\n",
"\n",
"Regardless of the number of layers, \n",
"the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of \n",
"non-linearity to the NN to be able to fit non-linear functions\n",
"Typical examples are the logistic *Sigmoid*"
"Regardless of the number of layers, the output of the NN will be\n",
"nothing but a linear function of the inputs. Thus we need to introduce\n",
"some kind of non-linearity to the NN to be able to fit non-linear\n",
"functions Typical examples are the logistic *Sigmoid*"
]
},
{
@@ -579,11 +592,11 @@
"source": [
"### Relevance\n",
"\n",
"The *sigmoid* function are more biologically plausible because \n",
"the output of inactive neurons are zero. Such activation function are called *one-sided*. However,\n",
"it has been shown that the hyperbolic tangent \n",
"performs better than the sigmoid for training MLPs. \n",
"has become the most popular for *deep neural networks*"
"The *sigmoid* function are more biologically plausible because the\n",
"output of inactive neurons are zero. Such activation function are\n",
"called *one-sided*. However, it has been shown that the hyperbolic\n",
"tangent performs better than the sigmoid for training MLPs. has\n",
"become the most popular for *deep neural networks*"
]
},
{
doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/NeuralNet/pdf/NeuralNet-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/NeuralNet/pdf/NeuralNet-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf
BIN
View File
Binary file not shown.
doc/src/NeuralNet/NeuralNet.do.txt
+77 -53
View File
@@ -137,16 +137,13 @@ means that each node in the output layer has a linear activation function). The
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
a separate type of NN due the unusual activation functions.
Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
of how a fully-connected FFNN works, and how it can be used to interpolate data sets.
!split
===== Multilayer perceptrons =====
One use often so-called fully-connected feed-forward neural networks with three
or more layers (an input layer, one or more hidden layers and an output layer)
consisting of neurons that have non-linear activation functions.
One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.
Such networks are often called *multilayer perceptrons* (MLPs)
@@ -156,16 +153,20 @@ Such networks are often called *multilayer perceptrons* (MLPs)
According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing
a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
assuming the activation function for the hidden layer is a _non-constant, bounded and monotonically-increasing continuous function_.
Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
assumed to be linear, so as to not restrict the range of output values.
We note that this theorem is only applicable to a NN with *one* hidden layer.
Therefore, we can easily construct an NN
that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
with activation functions that *do*. Furthermore, although the universal approximation theorem
lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit *all other*
functions.
Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.
We note that this theorem is only applicable to an NN with *one* hidden
layer. Therefore, we can easily construct an NN that employs
activation functions which do not satisfy the above requirements, as
long as we have at least one layer with activation functions that
*do*. Furthermore, although the universal approximation theorem lays
the theoretical foundation for regression with neural networks, it
does not say anything about how things work in practice: A neural
network can still be able to approximate a given function reasonably
well without having the flexibility to fit *all other* functions.
@@ -178,9 +179,11 @@ functions.
label{artificialNeuron2}
\end{equation}
!et
In an FFNN of such neurons, the *inputs* $x_i$
are the *outputs* of the neurons in the preceding layer. Furthermore, an MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer.
In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
of the outputs of *all* neurons in the previous layer.
!split
===== Mathematical model =====
@@ -201,7 +204,9 @@ producing the output $y_i^1$ of all neurons in layer 1,
label{outputLayer1}
\end{equation}
!et
where we assume that all nodes in the same layer have identical activation functions, hence the notation $f_l$
where we assume that all nodes in the same layer have identical
activation functions, hence the notation $f_l$
!bt
\begin{equation}
@@ -209,8 +214,11 @@ where we assume that all nodes in the same layer have identical activation funct
label{generalLayer}
\end{equation}
!et
where $N_l$ is the number of nodes in layer $l$. When the output of all the nodes in the first hidden layer are computed,
the values of the subsequent layer can be calculated and so forth until the output is obtained.
where $N_l$ is the number of nodes in layer $l$. When the output of
all the nodes in the first hidden layer are computed, the values of
the subsequent layer can be calculated and so forth until the output
is obtained.
@@ -239,8 +247,9 @@ where we have substituted $y_m^1$ with. Finally, the NN output yields,
!split
===== Mathematical model =====
We can generalize this expression to an MLP with $l$ hidden layers. The complete functional form
is,
We can generalize this expression to an MLP with $l$ hidden
layers. The complete functional form is,
!bt
\begin{align}
&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\!
@@ -250,31 +259,39 @@ is,
label{completeNN}
\end{align}
!et
which illustrates a basic property of MLPs: The only independent variables are the input values $x_n$.
which illustrates a basic property of MLPs: The only independent
variables are the input values $x_n$.
!split
===== Mathematical model =====
This confirms that an MLP,
despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors $\vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m$.
In our example, $n=2$ and $m=1$. Consequentially,
the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP.
This confirms that an MLP, despite its quite convoluted mathematical
form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors $\vec{x} \in \mathbb{R}^n \rightarrow
\vec{y} \in \mathbb{R}^m$. In our example, $n=2$ and
$m=1$. Consequentially, the number of input and output values of the
function we want to fit must be equal to the number of inputs and
outputs of our MLP.
Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that
the expression is essentially a nested sum of scaled activation functions of the form
Furthermore, the flexibility and universality of a MLP can be
illustrated by realizing that the expression is essentially a nested
sum of scaled activation functions of the form
!bt
\begin{equation}
h(x) = c_1 f(c_2 x + c_3) + c_4
\end{equation}
!et
where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
which is the key to the flexibility of a neural network.
where the parameters $c_i$ are weights and biases. By adjusting these
parameters, the activation functions can be shifted up and down or
left and right, change slope or be rescaled which is the key to the
flexibility of a neural network.
!split
=== Matrix-vector notation ===
We can introduce a more convenient notation for the activations in a NN.
Additionally, we can represent the biases and activations
@@ -307,6 +324,7 @@ the equation for the activations of hidden layer 2 in
!split
=== Matrix-vector notation and activation ===
The activation of node $i$ in layer 2 is
!bt
@@ -315,19 +333,22 @@ The activation of node $i$ in layer 2 is
f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
\end{equation}
!et
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
$\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer.
This is not just a convenient and compact notation, but also a useful
and intuitive way to think about MLPs: The output is calculated by a
series of matrix-vector multiplications and vector additions that are
used as input to the activation functions. For each operation
$\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer.
!split
=== Activation functions ===
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
As described in, the following restrictions are imposed on an activation function for a FFNN
to fulfill the universal approximation theorem
A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
* Non-constant
@@ -340,13 +361,15 @@ to fulfill the universal approximation theorem
!split
=== Activation functions, Logistic and Hyperbolic ones ===
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
layer simply performs a linear transformation of its inputs.
The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.
Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic *Sigmoid*
Regardless of the number of layers,
the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
non-linearity to the NN to be able to fit non-linear functions
Typical examples are the logistic *Sigmoid*
!bt
\begin{equation}
f(x) = \frac{1}{1 + e^{-x}},
@@ -363,11 +386,12 @@ and the *hyperbolic tangent* function
!split
=== Relevance ===
The *sigmoid* function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called *one-sided*. However,
it has been shown that the hyperbolic tangent
performs better than the sigmoid for training MLPs.
has become the most popular for *deep neural networks*
The *sigmoid* function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called *one-sided*. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs. has
become the most popular for *deep neural networks*
!bc pycod
"""The sigmoid function (or the logistic curve) is a