From b59bb722d3d6168ee40141945801ebd3dc77331c Mon Sep 17 00:00:00 2001 From: mhjensen Date: Fri, 28 Sep 2018 11:24:22 +0200 Subject: [PATCH] cleaning up --- doc/pub/NeuralNet/html/NeuralNet-bs.html | 129 ++++++++++------- doc/pub/NeuralNet/html/NeuralNet-reveal.html | 129 ++++++++++------- .../NeuralNet/html/NeuralNet-solarized.html | 129 ++++++++++------- doc/pub/NeuralNet/html/NeuralNet.html | 129 ++++++++++------- doc/pub/NeuralNet/ipynb/NeuralNet.ipynb | 119 +++++++++------- .../ipynb/ipynb-NeuralNet-src.tar.gz | Bin 209 -> 211 bytes .../pdf/NeuralNet-beamer-handouts2x3.pdf | Bin 286587 -> 286466 bytes doc/pub/NeuralNet/pdf/NeuralNet-beamer.pdf | Bin 269549 -> 269418 bytes doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf | Bin 350200 -> 349881 bytes doc/src/NeuralNet/NeuralNet.do.txt | 130 +++++++++++------- 10 files changed, 451 insertions(+), 314 deletions(-) diff --git a/doc/pub/NeuralNet/html/NeuralNet-bs.html b/doc/pub/NeuralNet/html/NeuralNet-bs.html index a38909828..6ca9b245b 100644 --- a/doc/pub/NeuralNet/html/NeuralNet-bs.html +++ b/doc/pub/NeuralNet/html/NeuralNet-bs.html @@ -322,19 +322,16 @@ means that each node in the output layer has a linear activation function). The there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as a separate type of NN due the unusual activation functions. -

-Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description -of how a fully-connected FFNN works, and how it can be used to interpolate data sets. -

Multilayer perceptrons

-One use often so-called fully-connected feed-forward neural networks with three -or more layers (an input layer, one or more hidden layers and an output layer) -consisting of neurons that have non-linear activation functions. +One uses often so-called fully-connected feed-forward neural networks +with three or more layers (an input layer, one or more hidden layers +and an output layer) consisting of neurons that have non-linear +activation functions.

Such networks are often called multilayer perceptrons (MLPs) @@ -348,17 +345,22 @@ Such networks are often called multilayer perceptrons (MLPs) According to the Universal approximation theorem, a feed-forward neural network with just a single hidden layer containing a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, assuming the activation function for the hidden layer is a non-constant, bounded and monotonically-increasing continuous function. -Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always -assumed to be linear, so as to not restrict the range of output values.

-We note that this theorem is only applicable to a NN with one hidden layer. -Therefore, we can easily construct an NN -that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer -with activation functions that do. Furthermore, although the universal approximation theorem -lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: -A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit all other -functions. +Note that the requirements on the activation function only applies to +the hidden layer, the output nodes are always assumed to be linear, so +as to not restrict the range of output values. + +

+We note that this theorem is only applicable to an NN with one hidden +layer. Therefore, we can easily construct an NN that employs +activation functions which do not satisfy the above requirements, as +long as we have at least one layer with activation functions that +do. Furthermore, although the universal approximation theorem lays +the theoretical foundation for regression with neural networks, it +does not say anything about how things work in practice: A neural +network can still be able to approximate a given function reasonably +well without having the flexibility to fit all other functions.

@@ -372,9 +374,11 @@ $$ \end{equation} $$ -In an FFNN of such neurons, the inputs \( x_i \) -are the outputs of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, -which means that each neuron receives a weighted sum of the outputs of all neurons in the previous layer. +

+In an FFNN of such neurons, the inputs \( x_i \) are the outputs of +the neurons in the preceding layer. Furthermore, an MLP is +fully-connected, which means that each neuron receives a weighted sum +of the outputs of all neurons in the previous layer.

@@ -401,7 +405,9 @@ $$ \end{equation} $$ -where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \) +

+where we assume that all nodes in the same layer have identical +activation functions, hence the notation \( f_l \) $$ \begin{equation} @@ -410,8 +416,11 @@ $$ \end{equation} $$ -where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed, -the values of the subsequent layer can be calculated and so forth until the output is obtained. +

+where \( N_l \) is the number of nodes in layer \( l \). When the output of +all the nodes in the first hidden layer are computed, the values of +the subsequent layer can be calculated and so forth until the output +is obtained.

@@ -448,8 +457,9 @@ $$

Mathematical model

-We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form -is, +We can generalize this expression to an MLP with \( l \) hidden +layers. The complete functional form is, + $$ \begin{align} &y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\! @@ -460,7 +470,9 @@ $$ \end{align} $$ -which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \). +

+which illustrates a basic property of MLPs: The only independent +variables are the input values \( x_n \).

@@ -468,15 +480,18 @@ which illustrates a basic property of MLPs: The only independent variables are t

Mathematical model

-This confirms that an MLP, -despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a -mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \). -In our example, \( n=2 \) and \( m=1 \). Consequentially, -the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. +This confirms that an MLP, despite its quite convoluted mathematical +form, is nothing more than an analytic function, specifically a +mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow +\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and +\( m=1 \). Consequentially, the number of input and output values of the +function we want to fit must be equal to the number of inputs and +outputs of our MLP.

-Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that -the expression is essentially a nested sum of scaled activation functions of the form +Furthermore, the flexibility and universality of a MLP can be +illustrated by realizing that the expression is essentially a nested +sum of scaled activation functions of the form $$ \begin{equation} @@ -485,15 +500,18 @@ $$ \end{equation} $$ -where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions -can be shifted up and down or left and right, change slope or be rescaled -which is the key to the flexibility of a neural network. +

+where the parameters \( c_i \) are weights and biases. By adjusting these +parameters, the activation functions can be shifted up and down or +left and right, change slope or be rescaled which is the key to the +flexibility of a neural network.

Matrix-vector notation

+

We can introduce a more convenient notation for the activations in a NN.

@@ -532,6 +550,7 @@ $$

Matrix-vector notation and activation

+

The activation of node \( i \) in layer 2 is $$ @@ -542,9 +561,11 @@ $$ \end{equation} $$ -This is not just a convenient and compact notation, but also -a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications -and vector additions that are used as input to the activation functions. For each operation +

+This is not just a convenient and compact notation, but also a useful +and intuitive way to think about MLPs: The output is calculated by a +series of matrix-vector multiplications and vector additions that are +used as input to the activation functions. For each operation \( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.

@@ -553,9 +574,10 @@ and vector additions that are used as input to the activation functions. For eac

Activation functions

-A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). -As described in, the following restrictions are imposed on an activation function for a FFNN -to fulfill the universal approximation theorem +A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). As described +in, the following restrictions are imposed on an activation function +for a FFNN to fulfill the universal approximation theorem