It is assumed that there are no weights and -bias at the input layer, so \( P = \{ P_{\text{hidden}}, -P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in -the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} } +bias at the input layer, so \( P = \{ P_{\mathrm{hidden}}, +P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in +the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} } \times 2 \) matrix.
-The first column in \( P_{\text{hidden} } \) represents +The first column in \( P_{\mathrm{hidden} } \) represents the bias for each neuron in the hidden layer and the second column -represents the weigths for each neuron. If there are \( N_{\text{output} -} \) neurons in the output layer, then \( P_{\text{output}} \) is a -\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first +represents the weigths for each neuron. If there are \( N_{\mathrm{output} +} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a +\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first column represents the bias of each neuron and the remaining columns represents the weights to each neuron. diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs092.html b/doc/pub/NeuralNet/html/._NeuralNet-bs092.html index 8aecae5db..39d3ae459 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs092.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs092.html @@ -381,7 +381,7 @@ $$ or, in terms of weights and biases for each layer: $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\} +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\} $$
@@ -405,10 +405,10 @@ $$ $$
-or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \) +or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) $$
diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs093.html b/doc/pub/NeuralNet/html/._NeuralNet-bs093.html index 5ae4d269a..c54af6cb3 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs093.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs093.html @@ -380,8 +380,8 @@ extension of it into a deep neural network would (hopefully) be painless.
-For simplicity, we assume that the input is an array \( \vec x = -(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural +For simplicity, we assume that the input is an array +\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (19).
diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs094.html b/doc/pub/NeuralNet/html/._NeuralNet-bs094.html index feabd678d..99244eaac 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs094.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs094.html @@ -366,24 +366,24 @@ MathJax.Hub.Config({
-First, a feedforward of the inputs must be done. This means that \( \vec - x \) must be passed through an input layer, a hidden layer and a output +First, a feedforward of the inputs must be done. This means that \( \hat{x} \) +must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the - data any further. The input layer will consist of \( N_{\text{input} } \) + data any further. The input layer will consist of \( N_{\mathrm{input} } \) neurons, passing its element to each neuron in the hidden layer. The - number of neurons in the hidden layer will be \( N_{\text{hidden} } \). + number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
-For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) -and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron +For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \) +and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is: $$ \begin{aligned} -z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\ +z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\ &= \begin{pmatrix} -b_i^{\text{hidden}} & w_i^{\text{hidden}} +b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 \\ diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs095.html b/doc/pub/NeuralNet/html/._NeuralNet-bs095.html index 3d26b73dd..7a2b19125 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs095.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs095.html @@ -369,28 +369,28 @@ MathJax.Hub.Config({ The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector: $$ \begin{aligned} -\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\ +\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\ &= \begin{pmatrix} - b_i^{\text{hidden}} & w_i^{\text{hidden}} + b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ x_1 & x_2 & \dots & x_N \end{pmatrix} \\ -&= \vec{p}_{i, \text{hidden}}^T X +&= \hat{p}_{i, \mathrm{hidden}}^T X \end{aligned} $$
-It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row -in \( P_{\text{hidden} } \), which contains the weights for the neural +It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row +in \( P_{\mathrm{hidden} } \), which contains the weights for the neural network to minimize according to (19).
-After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \) +After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \) in the hidden layer, the vector will be sent to an activation function -\( a_i(\vec{z}) \). In this example, the sigmoid function has been used: +\( a_i(\hat{z}) \). In this example, the sigmoid function has been used: $$ f(z) = \frac{1}{1 + \exp{(-z)}}. diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs096.html b/doc/pub/NeuralNet/html/._NeuralNet-bs096.html index ebd8365fc..81744050f 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs096.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs096.html @@ -366,20 +366,20 @@ MathJax.Hub.Config({
-The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is: +The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is: $$ -\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big). +\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big). $$
-The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer. +The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
The output layer consist of one neuron in this case, and combines the output from each of the neurons in the hidden layers. The output layer combines the results from the hidden layer using some weights \( -w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case, +w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case, it is assumes that the number of neurons in the output layer is one.
@@ -389,36 +389,36 @@ layer described previously. $$ \begin{aligned} -z_{1,j}^{\text{output}} & = +z_{1,j}^{\mathrm{output}} & = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{\text{hidden}} +\hat{x}_j^{\mathrm{hidden}} \end{pmatrix} \end{aligned} $$
-Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: +Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: $$ -\vec{z}_{1}^{\text{output}} = +\hat{z}_{1}^{\mathrm{output}} = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}} +\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}} \end{pmatrix} $$
In this case we seek a continous range of values since we are approximating a function. This means that after computing -\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its -feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final +\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its +feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final output of the network.
diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs099.html b/doc/pub/NeuralNet/html/._NeuralNet-bs099.html index 1ae4e033a..3d60f368d 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs099.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs099.html @@ -375,12 +375,12 @@ function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), +\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), $$
for a number of iterations or until $ \big|\big| \vec -\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some +\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some given tolerance.
diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs100.html b/doc/pub/NeuralNet/html/._NeuralNet-bs100.html index 9a5050094..4d5df7946 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs100.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs100.html @@ -368,17 +368,17 @@ MathJax.Hub.Config({
In our case, we have to minimize the cost function \( c(x, P) \) with respect to the two sets of weights and bisases, that is for the hidden -layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output} +layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output} } \) .
-This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is +This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P) +P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\ +P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P) \end{aligned} $$ diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs101.html b/doc/pub/NeuralNet/html/._NeuralNet-bs101.html index d64565b28..39d3c2082 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs101.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs101.html @@ -368,13 +368,13 @@ MathJax.Hub.Config({
As previously stated, a Deep Neural Network (DNN) follows the same concept of a neural network, but having more than one hidden -layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers -where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The +layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers +where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The input is still assumed to be an array of size \( 1 \times N \). The network must now try to optimalize its output with respect to the -collection of weigths and biases \( P = \big\{P_{\text{input} }, \ -P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \ -P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \). +collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \ +P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \ +P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs102.html b/doc/pub/NeuralNet/html/._NeuralNet-bs102.html index 8c0d86876..68b801209 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs102.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs102.html @@ -370,40 +370,40 @@ The feedforward step is similar to as for the neural netowork, but now consideri
The \( i \)-th neuron at layer \( l \) recieves the result -\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer \( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in -\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec -w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are -elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias -\( b_i^{(l), \ \text{hidden} } \): +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec +w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are +elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias +\( b_i^{(l), \ \mathrm{hidden} } \): $$ \begin{aligned} -z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\ +z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{(l-1),\text{hidden} } +\hat{x}_j^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} $$
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \): +The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \): $$ \begin{aligned} -\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\ +\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } +\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} $$ diff --git a/doc/pub/NeuralNet/html/NeuralNet-reveal.html b/doc/pub/NeuralNet/html/NeuralNet-reveal.html index 6e4a8b736..da0e17bf7 100644 --- a/doc/pub/NeuralNet/html/NeuralNet-reveal.html +++ b/doc/pub/NeuralNet/html/NeuralNet-reveal.html @@ -3172,7 +3172,7 @@ depth respectively). The neurons in a layer will only be connected to a small region of the layer before it, instead of all of the neurons in a fully-connected manner. Moreover, the final -output layer could for this specific image have dimensions \( 1\times 1 times 10 \), +output layer could for this specific image have dimensions \( 1\times 1 \times 10 \), because by the end of the CNN architecture we will reduce the full image into a single vector of class scores, arranged along the depth @@ -3821,17 +3821,17 @@ and biases for each layer.
It is assumed that there are no weights and -bias at the input layer, so \( P = \{ P_{\text{hidden}}, -P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in -the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} } +bias at the input layer, so \( P = \{ P_{\mathrm{hidden}}, +P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in +the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} } \times 2 \) matrix.
-The first column in \( P_{\text{hidden} } \) represents +The first column in \( P_{\mathrm{hidden} } \) represents the bias for each neuron in the hidden layer and the second column -represents the weigths for each neuron. If there are \( N_{\text{output} -} \) neurons in the output layer, then \( P_{\text{output}} \) is a -\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first +represents the weigths for each neuron. If there are \( N_{\mathrm{output} +} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a +\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first column represents the bias of each neuron and the remaining columns represents the weights to each neuron. @@ -3914,7 +3914,7 @@ or, in terms of weights and biases for each layer:
$$
-\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
+\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
$$
@@ -3943,11 +3943,11 @@ $$
-or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \) +or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \)
$$
-\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
+\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
$$
@@ -3971,8 +3971,8 @@ extension of it into a deep neural network would (hopefully) be
painless.
-For simplicity, we assume that the input is an array \( \vec x = -(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural +For simplicity, we assume that the input is an array +\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (19). @@ -3981,25 +3981,25 @@ network should find \( P \) such that it fulfills (19)
-First, a feedforward of the inputs must be done. This means that \( \vec - x \) must be passed through an input layer, a hidden layer and a output +First, a feedforward of the inputs must be done. This means that \( \hat{x} \) +must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the - data any further. The input layer will consist of \( N_{\text{input} } \) + data any further. The input layer will consist of \( N_{\mathrm{input} } \) neurons, passing its element to each neuron in the hidden layer. The - number of neurons in the hidden layer will be \( N_{\text{hidden} } \). + number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
-For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) -and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron +For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \) +and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is:
$$
\begin{aligned}
-z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
+z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
&=
\begin{pmatrix}
-b_i^{\text{hidden}} & w_i^{\text{hidden}}
+b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 \\
@@ -4019,29 +4019,29 @@ The result after weighting the input at the \( i \)-th hidden neuron can be writ
$$
\begin{aligned}
-\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
+\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
&=
\begin{pmatrix}
- b_i^{\text{hidden}} & w_i^{\text{hidden}}
+ b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
x_1 & x_2 & \dots & x_N
\end{pmatrix} \\
-&= \vec{p}_{i, \text{hidden}}^T X
+&= \hat{p}_{i, \mathrm{hidden}}^T X
\end{aligned}
$$
-It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row -in \( P_{\text{hidden} } \), which contains the weights for the neural +It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row +in \( P_{\mathrm{hidden} } \), which contains the weights for the neural network to minimize according to (19).
-After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \) +After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \) in the hidden layer, the vector will be sent to an activation function -\( a_i(\vec{z}) \). In this example, the sigmoid function has been used: +\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
$$
@@ -4055,22 +4055,22 @@ $$
-The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is: +The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
$$
-\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
+\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
$$
-The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer. +The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
The output layer consist of one neuron in this case, and combines the output from each of the neurons in the hidden layers. The output layer combines the results from the hidden layer using some weights \( -w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case, +w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case, it is assumes that the number of neurons in the output layer is one.
@@ -4081,30 +4081,30 @@ layer described previously.
$$
\begin{aligned}
-z_{1,j}^{\text{output}} & =
+z_{1,j}^{\mathrm{output}} & =
\begin{pmatrix}
-b_1^{\text{output}} & \vec{w}_1^{\text{output}}
+b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 \\
-\vec{x}_j^{\text{hidden}}
+\hat{x}_j^{\mathrm{hidden}}
\end{pmatrix}
\end{aligned}
$$
-Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: +Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
$$
-\vec{z}_{1}^{\text{output}} =
+\hat{z}_{1}^{\mathrm{output}} =
\begin{pmatrix}
-b_1^{\text{output}} & \vec{w}_1^{\text{output}}
+b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
-\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
+\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
\end{pmatrix}
$$
@@ -4112,8 +4112,8 @@ $$
In this case we seek a continous range of values since we are approximating a function. This means that after computing -\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its -feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final +\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its +feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final output of the network. @@ -4244,13 +4244,13 @@ goes as follows:
$$
-\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
+\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
$$
for a number of iterations or until $ \big|\big| \vec -\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some +\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some given tolerance.
@@ -4267,18 +4267,18 @@ the elements in \( \vec \omega \).
In our case, we have to minimize the cost function \( c(x, P) \) with respect to the two sets of weights and bisases, that is for the hidden -layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output} +layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output} } \) .
-This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is +This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is updated by
$$
\begin{aligned}
-P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
-P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
+P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
+P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
\end{aligned}
$$
@@ -4333,13 +4333,13 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
As previously stated, a Deep Neural Network (DNN) follows the same concept of a neural network, but having more than one hidden -layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers -where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The +layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers +where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The input is still assumed to be an array of size \( 1 \times N \). The network must now try to optimalize its output with respect to the -collection of weigths and biases \( P = \big\{P_{\text{input} }, \ -P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \ -P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \). +collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \ +P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \ +P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \). @@ -4351,43 +4351,43 @@ The feedforward step is similar to as for the neural netowork, but now consideri
The \( i \)-th neuron at layer \( l \) recieves the result -\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer \( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in -\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec -w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are -elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias -\( b_i^{(l), \ \text{hidden} } \): +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec +w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are +elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias +\( b_i^{(l), \ \mathrm{hidden} } \):
$$
\begin{aligned}
-z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
+z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
&=
\begin{pmatrix}
-b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
+b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 \\
-\vec{x}_j^{(l-1),\text{hidden} }
+\hat{x}_j^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
$$
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \): +The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
$$
\begin{aligned}
-\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
+\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
&=
\begin{pmatrix}
-b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
+b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
-\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
+\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
$$
diff --git a/doc/pub/NeuralNet/html/NeuralNet-solarized.html b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
index ca1df75cf..bfbe755ff 100644
--- a/doc/pub/NeuralNet/html/NeuralNet-solarized.html
+++ b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
@@ -3072,7 +3072,7 @@ depth respectively).
The neurons in a layer will
only be connected to a small region of the layer before it, instead of
all of the neurons in a fully-connected manner. Moreover, the final
-output layer could for this specific image have dimensions \( 1\times 1 times 10 \),
+output layer could for this specific image have dimensions \( 1\times 1 \times 10 \),
because by the
end of the CNN architecture we will reduce the full image into a
single vector of class scores, arranged along the depth
@@ -3703,17 +3703,17 @@ and biases for each layer.
It is assumed that there are no weights and -bias at the input layer, so \( P = \{ P_{\text{hidden}}, -P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in -the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} } +bias at the input layer, so \( P = \{ P_{\mathrm{hidden}}, +P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in +the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} } \times 2 \) matrix.
-The first column in \( P_{\text{hidden} } \) represents +The first column in \( P_{\mathrm{hidden} } \) represents the bias for each neuron in the hidden layer and the second column -represents the weigths for each neuron. If there are \( N_{\text{output} -} \) neurons in the output layer, then \( P_{\text{output}} \) is a -\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first +represents the weigths for each neuron. If there are \( N_{\mathrm{output} +} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a +\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first column represents the bias of each neuron and the remaining columns represents the weights to each neuron. @@ -3789,7 +3789,7 @@ $$ or, in terms of weights and biases for each layer: $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\} +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\} $$
@@ -3813,10 +3813,10 @@ $$ $$
-or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \) +or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) $$
@@ -3839,8 +3839,8 @@ extension of it into a deep neural network would (hopefully) be painless.
-For simplicity, we assume that the input is an array \( \vec x = -(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural +For simplicity, we assume that the input is an array +\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{eq:min}.
@@ -3849,24 +3849,24 @@ network should find \( P \) such that it fulfills \eqref{eq:min}.
-First, a feedforward of the inputs must be done. This means that \( \vec - x \) must be passed through an input layer, a hidden layer and a output +First, a feedforward of the inputs must be done. This means that \( \hat{x} \) +must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the - data any further. The input layer will consist of \( N_{\text{input} } \) + data any further. The input layer will consist of \( N_{\mathrm{input} } \) neurons, passing its element to each neuron in the hidden layer. The - number of neurons in the hidden layer will be \( N_{\text{hidden} } \). + number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
-For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) -and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron +For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \) +and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is: $$ \begin{aligned} -z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\ +z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\ &= \begin{pmatrix} -b_i^{\text{hidden}} & w_i^{\text{hidden}} +b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 \\ @@ -3884,28 +3884,28 @@ $$ The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector: $$ \begin{aligned} -\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\ +\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\ &= \begin{pmatrix} - b_i^{\text{hidden}} & w_i^{\text{hidden}} + b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ x_1 & x_2 & \dots & x_N \end{pmatrix} \\ -&= \vec{p}_{i, \text{hidden}}^T X +&= \hat{p}_{i, \mathrm{hidden}}^T X \end{aligned} $$
-It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row -in \( P_{\text{hidden} } \), which contains the weights for the neural +It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row +in \( P_{\mathrm{hidden} } \), which contains the weights for the neural network to minimize according to \eqref{eq:min}.
-After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \) +After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \) in the hidden layer, the vector will be sent to an activation function -\( a_i(\vec{z}) \). In this example, the sigmoid function has been used: +\( a_i(\hat{z}) \). In this example, the sigmoid function has been used: $$ f(z) = \frac{1}{1 + \exp{(-z)}}. @@ -3917,20 +3917,20 @@ $$
-The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is: +The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is: $$ -\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big). +\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big). $$
-The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer. +The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
The output layer consist of one neuron in this case, and combines the output from each of the neurons in the hidden layers. The output layer combines the results from the hidden layer using some weights \( -w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case, +w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case, it is assumes that the number of neurons in the output layer is one.
@@ -3940,36 +3940,36 @@ layer described previously. $$ \begin{aligned} -z_{1,j}^{\text{output}} & = +z_{1,j}^{\mathrm{output}} & = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{\text{hidden}} +\hat{x}_j^{\mathrm{hidden}} \end{pmatrix} \end{aligned} $$
-Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: +Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: $$ -\vec{z}_{1}^{\text{output}} = +\hat{z}_{1}^{\mathrm{output}} = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}} +\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}} \end{pmatrix} $$
In this case we seek a continous range of values since we are approximating a function. This means that after computing -\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its -feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final +\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its +feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final output of the network.
@@ -4095,12 +4095,12 @@ function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), +\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), $$
for a number of iterations or until $ \big|\big| \vec -\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some +\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some given tolerance.
@@ -4117,17 +4117,17 @@ the elements in \( \vec \omega \).
In our case, we have to minimize the cost function \( c(x, P) \) with respect to the two sets of weights and bisases, that is for the hidden -layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output} +layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output} } \) .
-This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is +This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P) +P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\ +P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P) \end{aligned} $$ @@ -4180,13 +4180,13 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
As previously stated, a Deep Neural Network (DNN) follows the same concept of a neural network, but having more than one hidden -layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers -where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The +layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers +where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The input is still assumed to be an array of size \( 1 \times N \). The network must now try to optimalize its output with respect to the -collection of weigths and biases \( P = \big\{P_{\text{input} }, \ -P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \ -P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \). +collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \ +P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \ +P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
@@ -4198,40 +4198,40 @@ The feedforward step is similar to as for the neural netowork, but now consideri
The \( i \)-th neuron at layer \( l \) recieves the result -\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer \( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in -\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec -w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are -elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias -\( b_i^{(l), \ \text{hidden} } \): +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec +w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are +elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias +\( b_i^{(l), \ \mathrm{hidden} } \): $$ \begin{aligned} -z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\ +z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{(l-1),\text{hidden} } +\hat{x}_j^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} $$
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \): +The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \): $$ \begin{aligned} -\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\ +\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } +\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} $$ diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html index 680938c01..e4b6df910 100644 --- a/doc/pub/NeuralNet/html/NeuralNet.html +++ b/doc/pub/NeuralNet/html/NeuralNet.html @@ -3077,7 +3077,7 @@ depth respectively). The neurons in a layer will only be connected to a small region of the layer before it, instead of all of the neurons in a fully-connected manner. Moreover, the final -output layer could for this specific image have dimensions \( 1\times 1 times 10 \), +output layer could for this specific image have dimensions \( 1\times 1 \times 10 \), because by the end of the CNN architecture we will reduce the full image into a single vector of class scores, arranged along the depth @@ -3708,17 +3708,17 @@ and biases for each layer.
It is assumed that there are no weights and -bias at the input layer, so \( P = \{ P_{\text{hidden}}, -P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in -the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} } +bias at the input layer, so \( P = \{ P_{\mathrm{hidden}}, +P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in +the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} } \times 2 \) matrix.
-The first column in \( P_{\text{hidden} } \) represents +The first column in \( P_{\mathrm{hidden} } \) represents the bias for each neuron in the hidden layer and the second column -represents the weigths for each neuron. If there are \( N_{\text{output} -} \) neurons in the output layer, then \( P_{\text{output}} \) is a -\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first +represents the weigths for each neuron. If there are \( N_{\mathrm{output} +} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a +\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first column represents the bias of each neuron and the remaining columns represents the weights to each neuron. @@ -3794,7 +3794,7 @@ $$ or, in terms of weights and biases for each layer: $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\} +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\} $$
@@ -3818,10 +3818,10 @@ $$ $$
-or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \) +or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) $$
@@ -3844,8 +3844,8 @@ extension of it into a deep neural network would (hopefully) be painless.
-For simplicity, we assume that the input is an array \( \vec x = -(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural +For simplicity, we assume that the input is an array +\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{eq:min}.
@@ -3854,24 +3854,24 @@ network should find \( P \) such that it fulfills \eqref{eq:min}.
-First, a feedforward of the inputs must be done. This means that \( \vec - x \) must be passed through an input layer, a hidden layer and a output +First, a feedforward of the inputs must be done. This means that \( \hat{x} \) +must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the - data any further. The input layer will consist of \( N_{\text{input} } \) + data any further. The input layer will consist of \( N_{\mathrm{input} } \) neurons, passing its element to each neuron in the hidden layer. The - number of neurons in the hidden layer will be \( N_{\text{hidden} } \). + number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
-For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) -and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron +For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \) +and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is: $$ \begin{aligned} -z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\ +z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\ &= \begin{pmatrix} -b_i^{\text{hidden}} & w_i^{\text{hidden}} +b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 \\ @@ -3889,28 +3889,28 @@ $$ The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector: $$ \begin{aligned} -\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\ +\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\ &= \begin{pmatrix} - b_i^{\text{hidden}} & w_i^{\text{hidden}} + b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ x_1 & x_2 & \dots & x_N \end{pmatrix} \\ -&= \vec{p}_{i, \text{hidden}}^T X +&= \hat{p}_{i, \mathrm{hidden}}^T X \end{aligned} $$
-It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row -in \( P_{\text{hidden} } \), which contains the weights for the neural +It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row +in \( P_{\mathrm{hidden} } \), which contains the weights for the neural network to minimize according to \eqref{eq:min}.
-After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \) +After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \) in the hidden layer, the vector will be sent to an activation function -\( a_i(\vec{z}) \). In this example, the sigmoid function has been used: +\( a_i(\hat{z}) \). In this example, the sigmoid function has been used: $$ f(z) = \frac{1}{1 + \exp{(-z)}}. @@ -3922,20 +3922,20 @@ $$
-The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is: +The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is: $$ -\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big). +\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big). $$
-The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer. +The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
The output layer consist of one neuron in this case, and combines the output from each of the neurons in the hidden layers. The output layer combines the results from the hidden layer using some weights \( -w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case, +w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case, it is assumes that the number of neurons in the output layer is one.
@@ -3945,36 +3945,36 @@ layer described previously. $$ \begin{aligned} -z_{1,j}^{\text{output}} & = +z_{1,j}^{\mathrm{output}} & = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{\text{hidden}} +\hat{x}_j^{\mathrm{hidden}} \end{pmatrix} \end{aligned} $$
-Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: +Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: $$ -\vec{z}_{1}^{\text{output}} = +\hat{z}_{1}^{\mathrm{output}} = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}} +\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}} \end{pmatrix} $$
In this case we seek a continous range of values since we are approximating a function. This means that after computing -\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its -feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final +\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its +feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final output of the network.
@@ -4100,12 +4100,12 @@ function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), +\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), $$
for a number of iterations or until $ \big|\big| \vec -\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some +\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some given tolerance.
@@ -4122,17 +4122,17 @@ the elements in \( \vec \omega \).
In our case, we have to minimize the cost function \( c(x, P) \) with respect to the two sets of weights and bisases, that is for the hidden -layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output} +layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output} } \) .
-This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is +This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P) +P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\ +P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P) \end{aligned} $$ @@ -4185,13 +4185,13 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
As previously stated, a Deep Neural Network (DNN) follows the same concept of a neural network, but having more than one hidden -layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers -where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The +layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers +where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The input is still assumed to be an array of size \( 1 \times N \). The network must now try to optimalize its output with respect to the -collection of weigths and biases \( P = \big\{P_{\text{input} }, \ -P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \ -P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \). +collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \ +P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \ +P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
@@ -4203,40 +4203,40 @@ The feedforward step is similar to as for the neural netowork, but now consideri
The \( i \)-th neuron at layer \( l \) recieves the result -\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer \( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in -\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec -w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are -elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias -\( b_i^{(l), \ \text{hidden} } \): +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec +w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are +elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias +\( b_i^{(l), \ \mathrm{hidden} } \): $$ \begin{aligned} -z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\ +z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{(l-1),\text{hidden} } +\hat{x}_j^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} $$
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \): +The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \): $$ \begin{aligned} -\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\ +\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } +\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} $$ diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb index 61d8d5658..07dc89694 100644 --- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb +++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb @@ -3378,7 +3378,7 @@ "The neurons in a layer will\n", "only be connected to a small region of the layer before it, instead of\n", "all of the neurons in a fully-connected manner. Moreover, the final\n", - "output layer could for this specific image have dimensions $1\\times 1 times 10$, \n", + "output layer could for this specific image have dimensions $1\\times 1 \\times 10$, \n", "because by the\n", "end of the CNN architecture we will reduce the full image into a\n", "single vector of class scores, arranged along the depth\n", @@ -4071,16 +4071,16 @@ "and biases for each layer. \n", "\n", "It is assumed that there are no weights and\n", - "bias at the input layer, so $P = \\{ P_{\\text{hidden}},\n", - "P_{\\text{output}} \\}$. If there are $N_{\\text{hidden} }$ neurons in\n", - "the hidden layer, then $P_{\\text{hidden}}$ is an $N_{\\text{hidden} }\n", + "bias at the input layer, so $P = \\{ P_{\\mathrm{hidden}},\n", + "P_{\\mathrm{output}} \\}$. If there are $N_{\\mathrm{hidden} }$ neurons in\n", + "the hidden layer, then $P_{\\mathrm{hidden}}$ is an $N_{\\mathrm{hidden} }\n", "\\times 2$ matrix. \n", "\n", - "The first column in $P_{\\text{hidden} }$ represents\n", + "The first column in $P_{\\mathrm{hidden} }$ represents\n", "the bias for each neuron in the hidden layer and the second column\n", - "represents the weigths for each neuron. If there are $N_{\\text{output}\n", - "}$ neurons in the output layer, then $P_{\\text{output}} $ is a\n", - "$N_{\\text{output} } \\times (1 + N_{\\text{hidden} })$ matrix. Its first\n", + "represents the weigths for each neuron. If there are $N_{\\mathrm{output}\n", + "}$ neurons in the output layer, then $P_{\\mathrm{output}} $ is a\n", + "$N_{\\mathrm{output} } \\times (1 + N_{\\mathrm{hidden} })$ matrix. Its first\n", "column represents the bias of each neuron and the remaining columns\n", "represents the weights to each neuron.\n", "\n", @@ -4183,7 +4183,7 @@ "metadata": {}, "source": [ "$$\n", - "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) \\big)^2 \\Big\\}\n", + "\\min_{P_{\\mathrm{hidden} }, \\ P_{\\mathrm{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\mathrm{hidden} }, P_{\\mathrm{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\mathrm{hidden} }, P_{\\mathrm{output} }\\}) \\big)^2 \\Big\\}\n", "$$" ] }, @@ -4232,7 +4232,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "or in terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$" + "or in terms of $P_{\\mathrm{hidden} }$ and $P_{\\mathrm{output} }$" ] }, { @@ -4240,7 +4240,7 @@ "metadata": {}, "source": [ "$$\n", - "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(x, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n", + "\\min_{P_{\\mathrm{hidden} }, \\ P_{\\mathrm{output} }} c(x, \\{P_{\\mathrm{hidden} }, P_{\\mathrm{output} }\\})\n", "$$" ] }, @@ -4263,21 +4263,21 @@ "extension of it into a deep neural network would (hopefully) be\n", "painless.\n", "\n", - "For simplicity, we assume that the input is an array $\\vec x =\n", - "(x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural\n", + "For simplicity, we assume that the input is an array \n", + "$\\hat{x}= (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural\n", "network should find $P$ such that it fulfills ([eq:min](#eq:min)).\n", "\n", "## Feedforward\n", "\n", - "First, a feedforward of the inputs must be done. This means that $\\vec\n", - " x$ must be passed through an input layer, a hidden layer and a output\n", + "First, a feedforward of the inputs must be done. This means that $\\hat{x}$ \n", + "must be passed through an input layer, a hidden layer and a output\n", " layer. The input layer in this case, does not need to process the\n", - " data any further. The input layer will consist of $N_{\\text{input} }$\n", + " data any further. The input layer will consist of $N_{\\mathrm{input} }$\n", " neurons, passing its element to each neuron in the hidden layer. The\n", - " number of neurons in the hidden layer will be $N_{\\text{hidden} }$.\n", + " number of neurons in the hidden layer will be $N_{\\mathrm{hidden} }$.\n", "\n", - "For the $i$-th in the hidden layer with weight $w_i^{\\text{hidden} }$\n", - "and bias $b_i^{\\text{hidden} }$, the weighting from the $j$-th neuron\n", + "For the $i$-th in the hidden layer with weight $w_i^{\\mathrm{hidden} }$\n", + "and bias $b_i^{\\mathrm{hidden} }$, the weighting from the $j$-th neuron\n", "at the input layer is:" ] }, @@ -4287,10 +4287,10 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "z_{i,j}^{\\text{hidden}} &= b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_j \\\\\n", + "z_{i,j}^{\\mathrm{hidden}} &= b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}}x_j \\\\\n", "&= \n", "\\begin{pmatrix}\n", - "b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n", + "b_i^{\\mathrm{hidden}} & w_i^{\\mathrm{hidden}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 \\\\\n", @@ -4316,16 +4316,16 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "\\vec{z}_{i}^{\\text{hidden}} &= \\Big( b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_1 , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_2, \\ \\dots \\, , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_N\\Big) \\\\\n", + "\\hat{z}_{i}^{\\mathrm{hidden}} &= \\Big( b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}}x_1 , \\ b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}} x_2, \\ \\dots \\, , \\ b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}} x_N\\Big) \\\\\n", "&= \n", "\\begin{pmatrix}\n", - " b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n", + " b_i^{\\mathrm{hidden}} & w_i^{\\mathrm{hidden}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 & 1 & \\dots & 1 \\\\\n", "x_1 & x_2 & \\dots & x_N\n", "\\end{pmatrix} \\\\\n", - "&= \\vec{p}_{i, \\text{hidden}}^T X\n", + "&= \\hat{p}_{i, \\mathrm{hidden}}^T X\n", "\\end{aligned}\n", "$$" ] @@ -4334,13 +4334,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It is the vector $\\vec{p}_{i, \\text{hidden}}^T$ that defines each row\n", - "in $P_{\\text{hidden} }$, which contains the weights for the neural\n", + "It is the vector $\\hat{p}_{i, \\mathrm{hidden}}^T$ that defines each row\n", + "in $P_{\\mathrm{hidden} }$, which contains the weights for the neural\n", "network to minimize according to ([eq:min](#eq:min)).\n", "\n", - "After having found $\\vec{z}_{i}^{\\text{hidden}} $ for every neuron $i$\n", + "After having found $\\hat{z}_{i}^{\\mathrm{hidden}} $ for every neuron $i$\n", "in the hidden layer, the vector will be sent to an activation function\n", - "$a_i(\\vec{z})$. In this example, the sigmoid function has been used:\n", + "$a_i(\\hat{z})$. In this example, the sigmoid function has been used:\n", "\n", "$$\n", "f(z) = \\frac{1}{1 + \\exp{(-z)}}.\n", @@ -4349,18 +4349,18 @@ "\n", "## Output\n", "\n", - "The output $\\vec{x}_i^{\\text{hidden} }$from each $i$-th hidden neuron is:\n", + "The output $\\hat{x}_i^{\\mathrm{hidden}}$from each $i$-th hidden neuron is:\n", "\n", "$$\n", - "\\vec{x}_i^{\\text{hidden} } = f\\big( \\vec{z}_{i}^{\\text{hidden}} \\big).\n", + "\\hat{x}_i^{\\mathrm{hidden} } = f\\big( \\hat{z}_{i}^{\\mathrm{hidden}} \\big).\n", "$$\n", "\n", - "The outputs $\\vec{x}_i^{\\text{hidden} } $ are then sent to the output layer. \n", + "The outputs $\\hat{x}_i^{\\mathrm{hidden} } $ are then sent to the output layer. \n", "\n", "The output layer consist of one neuron in this case, and combines the\n", "output from each of the neurons in the hidden layers. The output layer\n", "combines the results from the hidden layer using some weights $\n", - "w_i^{\\text{output}}$ and biases $b_i^{\\text{output}}$. In this case,\n", + "w_i^{\\mathrm{output}}$ and biases $b_i^{\\mathrm{output}}$. In this case,\n", "it is assumes that the number of neurons in the output layer is one.\n", "\n", "The procedure of weigthing the output neuron $j$ in the hidden layer\n", @@ -4374,13 +4374,13 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "z_{1,j}^{\\text{output}} & = \n", + "z_{1,j}^{\\mathrm{output}} & = \n", "\\begin{pmatrix}\n", - "b_1^{\\text{output}} & \\vec{w}_1^{\\text{output}}\n", + "b_1^{\\mathrm{output}} & \\hat{w}_1^{\\mathrm{output}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 \\\\\n", - "\\vec{x}_j^{\\text{hidden}}\n", + "\\hat{x}_j^{\\mathrm{hidden}}\n", "\\end{pmatrix}\n", "\\end{aligned}\n", "$$" @@ -4390,23 +4390,23 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Expressing $z_{1,j}^{\\text{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:\n", + "Expressing $z_{1,j}^{\\mathrm{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:\n", "\n", "$$\n", - "\\vec{z}_{1}^{\\text{output}} = \n", + "\\hat{z}_{1}^{\\mathrm{output}} = \n", "\\begin{pmatrix}\n", - "b_1^{\\text{output}} & \\vec{w}_1^{\\text{output}}\n", + "b_1^{\\mathrm{output}} & \\hat{w}_1^{\\mathrm{output}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 & 1 & \\dots & 1 \\\\\n", - "\\vec{x}_1^{\\text{hidden}} & \\vec{x}_2^{\\text{hidden}} & \\dots & \\vec{x}_N^{\\text{hidden}}\n", + "\\hat{x}_1^{\\mathrm{hidden}} & \\hat{x}_2^{\\mathrm{hidden}} & \\dots & \\hat{x}_N^{\\mathrm{hidden}}\n", "\\end{pmatrix}\n", "$$\n", "\n", "In this case we seek a continous range of values since we are\n", "approximating a function. This means that after computing\n", - "$\\vec{z}_{1}^{\\text{output}}$ the neural network has finished its\n", - "feedforward step, and $\\vec{z}_{1}^{\\text{output}}$ is the final\n", + "$\\hat{z}_{1}^{\\mathrm{output}}$ the neural network has finished its\n", + "feedforward step, and $\\hat{z}_{1}^{\\mathrm{output}}$ is the final\n", "output of the network.\n", "\n", "\n", @@ -4555,7 +4555,7 @@ "metadata": {}, "source": [ "$$\n", - "\\vec \\omega_{\\text{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(x, \\vec \\omega),\n", + "\\vec \\omega_{\\mathrm{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(x, \\vec \\omega),\n", "$$" ] }, @@ -4564,7 +4564,7 @@ "metadata": {}, "source": [ "for a number of iterations or until $ \\big|\\big| \\vec\n", - "\\omega_{\\text{new} } - \\vec \\omega \\big|\\big|$ is smaller than some\n", + "\\omega_{\\mathrm{new} } - \\vec \\omega \\big|\\big|$ is smaller than some\n", "given tolerance.\n", "\n", "The value of $\\lambda$ decides how large steps the algorithm must take\n", @@ -4577,10 +4577,10 @@ "\n", "In our case, we have to minimize the cost function $c(x, P)$ with\n", "respect to the two sets of weights and bisases, that is for the hidden\n", - "layer $P_{\\text{hidden} }$ and for the ouput layer $P_{\\text{output}\n", + "layer $P_{\\mathrm{hidden} }$ and for the ouput layer $P_{\\mathrm{output}\n", "}$ .\n", "\n", - "This means that $P_{\\text{hidden} }$ and $P_{\\text{output} }$ is\n", + "This means that $P_{\\mathrm{hidden} }$ and $P_{\\mathrm{output} }$ is\n", "updated by" ] }, @@ -4590,8 +4590,8 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(x, P) \\\\\n", - "P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(x, P) \n", + "P_{\\mathrm{hidden},\\mathrm{new}} &= P_{\\mathrm{hidden}} - \\lambda \\nabla_{P_{\\mathrm{hidden}}} c(x, P) \\\\\n", + "P_{\\mathrm{output},\\mathrm{new}} &= P_{\\mathrm{output}} - \\lambda \\nabla_{P_{\\mathrm{output}}} c(x, P) \n", "\\end{aligned}\n", "$$" ] @@ -4655,13 +4655,13 @@ "\n", "As previously stated, a Deep Neural Network (DNN) follows the same\n", "concept of a neural network, but having more than one hidden\n", - "layer. Suppose that the network has $N_{\\text{hidden}}$ hidden layers\n", - "where the $l$-th layer has $N_{\\text{hidden}}^{(l)}$ neurons. The\n", + "layer. Suppose that the network has $N_{\\mathrm{hidden}}$ hidden layers\n", + "where the $l$-th layer has $N_{\\mathrm{hidden}}^{(l)}$ neurons. The\n", "input is still assumed to be an array of size $1 \\times N$. The\n", "network must now try to optimalize its output with respect to the\n", - "collection of weigths and biases $P = \\big\\{P_{\\text{input} }, \\\n", - "P_{\\text{hidden} }^{(1)}, \\ P_{\\text{hidden} }^{(2)}, \\ \\dots , \\\n", - "P_{\\text{hidden} }^{(N_{\\text{hidden}})}, \\ P_{\\text{output} }\\big\\}$.\n", + "collection of weigths and biases $P = \\big\\{P_{\\mathrm{input} }, \\\n", + "P_{\\mathrm{hidden} }^{(1)}, \\ P_{\\mathrm{hidden} }^{(2)}, \\ \\dots , \\\n", + "P_{\\mathrm{hidden} }^{(N_{\\mathrm{hidden}})}, \\ P_{\\mathrm{output} }\\big\\}$.\n", "\n", "\n", "## Feed forward again\n", @@ -4669,12 +4669,12 @@ "The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer. \n", "\n", "The $i$-th neuron at layer $l$ recieves the result\n", - "$\\vec{x}_j^{(l-1),\\text{hidden} }$ from the $j$-th neuron at layer\n", + "$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$ from the $j$-th neuron at layer\n", "$l-1$. The $i$-th neuron at layer $l$ weights all of the elements in\n", - "$\\vec{x}_j^{(l-1),\\text{hidden} }$ with a weight vector $\\vec\n", - "w_{i,j}^{(l), \\ \\text{hidden} }$ with as many weigths as there are\n", - "elements in$\\vec{x}_j^{(l-1),\\text{hidden} }$, and adds a bias\n", - "$b_i^{(l), \\ \\text{hidden} }$:" + "$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$ with a weight vector $\\vec\n", + "w_{i,j}^{(l), \\ \\mathrm{hidden} }$ with as many weigths as there are\n", + "elements in$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$, and adds a bias\n", + "$b_i^{(l), \\ \\mathrm{hidden} }$:" ] }, { @@ -4683,14 +4683,14 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "z_{i,j}^{(l),\\ \\text{hidden}} &= b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_j^{(l-1),\\text{hidden} } \\\\\n", + "z_{i,j}^{(l),\\ \\mathrm{hidden}} &= b_i^{(l), \\ \\mathrm{hidden}} + \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\\hat{x}_j^{(l-1),\\mathrm{hidden} } \\\\\n", "&= \n", "\\begin{pmatrix}\n", - "b_i^{(l), \\ \\text{hidden}} & \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\n", + "b_i^{(l), \\ \\mathrm{hidden}} & \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 \\\\\n", - "\\vec{x}_j^{(l-1),\\text{hidden} }\n", + "\\hat{x}_j^{(l-1),\\mathrm{hidden} }\n", "\\end{pmatrix} \n", "\\end{aligned}\n", "$$" @@ -4700,7 +4700,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\\vec{z}_{i}^{(l),\\ \\text{hidden}}$:" + "The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\\hat{z}_{i}^{(l),\\ \\mathrm{hidden}}$:" ] }, { @@ -4709,14 +4709,14 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "\\vec{z}_{i}^{(l),\\ \\text{hidden}} &= \\Big( b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_1^{(l-1),\\text{hidden} }, \\ \\dots \\ , \\ b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\text{hidden} } \\Big) \\\\\n", + "\\hat{z}_{i}^{(l),\\ \\mathrm{hidden}} &= \\Big( b_i^{(l), \\ \\mathrm{hidden}} + \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\\hat{x}_1^{(l-1),\\mathrm{hidden} }, \\ \\dots \\ , \\ b_i^{(l), \\ \\mathrm{hidden}} + \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\mathrm{hidden} } \\Big) \\\\\n", "&= \n", "\\begin{pmatrix}\n", - "b_i^{(l), \\ \\text{hidden}} & \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\n", + "b_i^{(l), \\ \\mathrm{hidden}} & \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 & 1 & \\dots & 1 \\\\\n", - "\\vec{x}_{1}^{(l-1),\\text{hidden} } & \\vec{x}_{2}^{(l-1),\\text{hidden} } & \\dots & \\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\text{hidden} }\n", + "\\hat{x}_{1}^{(l-1),\\mathrm{hidden} } & \\hat{x}_{2}^{(l-1),\\mathrm{hidden} } & \\dots & \\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\mathrm{hidden} }\n", "\\end{pmatrix}\n", "\\end{aligned}\n", "$$" diff --git a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz index a3461374c..67b12625b 100644 Binary files a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz and b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz differ diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf index e6e3cc0f6..95ce82474 100644 Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf differ diff --git a/doc/src/NeuralNet/NeuralNet.do.txt b/doc/src/NeuralNet/NeuralNet.do.txt index 1bfb84e4b..2eb2616b9 100644 --- a/doc/src/NeuralNet/NeuralNet.do.txt +++ b/doc/src/NeuralNet/NeuralNet.do.txt @@ -2517,7 +2517,7 @@ depth respectively). The neurons in a layer will only be connected to a small region of the layer before it, instead of all of the neurons in a fully-connected manner. Moreover, the final -output layer could for this specific image have dimensions $1\times 1 times 10$, +output layer could for this specific image have dimensions $1\times 1 \times 10$, because by the end of the CNN architecture we will reduce the full image into a single vector of class scores, arranged along the depth @@ -3088,16 +3088,16 @@ neural network $N(x,P)$ with $P $ being the collection of the weights and biases for each layer. It is assumed that there are no weights and -bias at the input layer, so $P = \{ P_{\text{hidden}}, -P_{\text{output}} \}$. If there are $N_{\text{hidden} }$ neurons in -the hidden layer, then $P_{\text{hidden}}$ is an $N_{\text{hidden} } +bias at the input layer, so $P = \{ P_{\mathrm{hidden}}, +P_{\mathrm{output}} \}$. If there are $N_{\mathrm{hidden} }$ neurons in +the hidden layer, then $P_{\mathrm{hidden}}$ is an $N_{\mathrm{hidden} } \times 2$ matrix. -The first column in $P_{\text{hidden} }$ represents +The first column in $P_{\mathrm{hidden} }$ represents the bias for each neuron in the hidden layer and the second column -represents the weigths for each neuron. If there are $N_{\text{output} -}$ neurons in the output layer, then $P_{\text{output}} $ is a -$N_{\text{output} } \times (1 + N_{\text{hidden} })$ matrix. Its first +represents the weigths for each neuron. If there are $N_{\mathrm{output} +}$ neurons in the output layer, then $P_{\mathrm{output}} $ is a +$N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} })$ matrix. Its first column represents the bias of each neuron and the remaining columns represents the weights to each neuron. @@ -3163,7 +3163,7 @@ or, in terms of weights and biases for each layer: !bt \[ -\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\} +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\} \] !et @@ -3187,11 +3187,11 @@ our network must solve, is \] !et -or in terms of $P_{\text{hidden} }$ and $P_{\text{output} }$ +or in terms of $P_{\mathrm{hidden} }$ and $P_{\mathrm{output} }$ !bt \[ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \] !et @@ -3211,30 +3211,30 @@ network. Having an implementation of a neural network at hand, an extension of it into a deep neural network would (hopefully) be painless. -For simplicity, we assume that the input is an array $\vec x = -(x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural +For simplicity, we assume that the input is an array +$\hat{x}= (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{eq:min}). !split ===== Feedforward ===== -First, a feedforward of the inputs must be done. This means that $\vec - x$ must be passed through an input layer, a hidden layer and a output +First, a feedforward of the inputs must be done. This means that $\hat{x}$ +must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the - data any further. The input layer will consist of $N_{\text{input} }$ + data any further. The input layer will consist of $N_{\mathrm{input} }$ neurons, passing its element to each neuron in the hidden layer. The - number of neurons in the hidden layer will be $N_{\text{hidden} }$. + number of neurons in the hidden layer will be $N_{\mathrm{hidden} }$. -For the $i$-th in the hidden layer with weight $w_i^{\text{hidden} }$ -and bias $b_i^{\text{hidden} }$, the weighting from the $j$-th neuron +For the $i$-th in the hidden layer with weight $w_i^{\mathrm{hidden} }$ +and bias $b_i^{\mathrm{hidden} }$, the weighting from the $j$-th neuron at the input layer is: !bt \begin{aligned} -z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\ +z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\ &= \begin{pmatrix} -b_i^{\text{hidden}} & w_i^{\text{hidden}} +b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 \\ @@ -3250,26 +3250,26 @@ x_j The result after weighting the input at the $i$-th hidden neuron can be written as a vector: !bt \begin{aligned} -\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\ +\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\ &= \begin{pmatrix} - b_i^{\text{hidden}} & w_i^{\text{hidden}} + b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ x_1 & x_2 & \dots & x_N \end{pmatrix} \\ -&= \vec{p}_{i, \text{hidden}}^T X +&= \hat{p}_{i, \mathrm{hidden}}^T X \end{aligned} !et -It is the vector $\vec{p}_{i, \text{hidden}}^T$ that defines each row -in $P_{\text{hidden} }$, which contains the weights for the neural +It is the vector $\hat{p}_{i, \mathrm{hidden}}^T$ that defines each row +in $P_{\mathrm{hidden} }$, which contains the weights for the neural network to minimize according to (ref{eq:min}). -After having found $\vec{z}_{i}^{\text{hidden}} $ for every neuron $i$ +After having found $\hat{z}_{i}^{\mathrm{hidden}} $ for every neuron $i$ in the hidden layer, the vector will be sent to an activation function -$a_i(\vec{z})$. In this example, the sigmoid function has been used: +$a_i(\hat{z})$. In this example, the sigmoid function has been used: $$ f(z) = \frac{1}{1 + \exp{(-z)}}. @@ -3279,18 +3279,18 @@ $$ !split ===== Output ===== -The output $\vec{x}_i^{\text{hidden} }$from each $i$-th hidden neuron is: +The output $\hat{x}_i^{\mathrm{hidden}}$from each $i$-th hidden neuron is: $$ -\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big). +\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big). $$ -The outputs $\vec{x}_i^{\text{hidden} } $ are then sent to the output layer. +The outputs $\hat{x}_i^{\mathrm{hidden} } $ are then sent to the output layer. The output layer consist of one neuron in this case, and combines the output from each of the neurons in the hidden layers. The output layer combines the results from the hidden layer using some weights $ -w_i^{\text{output}}$ and biases $b_i^{\text{output}}$. In this case, +w_i^{\mathrm{output}}$ and biases $b_i^{\mathrm{output}}$. In this case, it is assumes that the number of neurons in the output layer is one. The procedure of weigthing the output neuron $j$ in the hidden layer @@ -3299,34 +3299,34 @@ layer described previously. !bt \begin{aligned} -z_{1,j}^{\text{output}} & = +z_{1,j}^{\mathrm{output}} & = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{\text{hidden}} +\hat{x}_j^{\mathrm{hidden}} \end{pmatrix} \end{aligned} !et -Expressing $z_{1,j}^{\text{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer: +Expressing $z_{1,j}^{\mathrm{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer: $$ -\vec{z}_{1}^{\text{output}} = +\hat{z}_{1}^{\mathrm{output}} = \begin{pmatrix} -b_1^{\text{output}} & \vec{w}_1^{\text{output}} +b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}} +\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}} \end{pmatrix} $$ In this case we seek a continous range of values since we are approximating a function. This means that after computing -$\vec{z}_{1}^{\text{output}}$ the neural network has finished its -feedforward step, and $\vec{z}_{1}^{\text{output}}$ is the final +$\hat{z}_{1}^{\mathrm{output}}$ the neural network has finished its +feedforward step, and $\hat{z}_{1}^{\mathrm{output}}$ is the final output of the network. @@ -3442,12 +3442,12 @@ goes as follows: !bt \[ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), +\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), \] !et for a number of iterations or until $ \big|\big| \vec -\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some +\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some given tolerance. The value of $\lambda$ decides how large steps the algorithm must take @@ -3461,16 +3461,16 @@ the elements in $\vec \omega$. In our case, we have to minimize the cost function $c(x, P)$ with respect to the two sets of weights and bisases, that is for the hidden -layer $P_{\text{hidden} }$ and for the ouput layer $P_{\text{output} +layer $P_{\mathrm{hidden} }$ and for the ouput layer $P_{\mathrm{output} }$ . -This means that $P_{\text{hidden} }$ and $P_{\text{output} }$ is +This means that $P_{\mathrm{hidden} }$ and $P_{\mathrm{output} }$ is updated by !bt \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P) +P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\ +P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P) \end{aligned} !et @@ -3519,13 +3519,13 @@ def solve_ode_neural_network(x, num_neurons_hidden, num_iter, lmb): As previously stated, a Deep Neural Network (DNN) follows the same concept of a neural network, but having more than one hidden -layer. Suppose that the network has $N_{\text{hidden}}$ hidden layers -where the $l$-th layer has $N_{\text{hidden}}^{(l)}$ neurons. The +layer. Suppose that the network has $N_{\mathrm{hidden}}$ hidden layers +where the $l$-th layer has $N_{\mathrm{hidden}}^{(l)}$ neurons. The input is still assumed to be an array of size $1 \times N$. The network must now try to optimalize its output with respect to the -collection of weigths and biases $P = \big\{P_{\text{input} }, \ -P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \ -P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\}$. +collection of weigths and biases $P = \big\{P_{\mathrm{input} }, \ +P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \ +P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\}$. !split @@ -3534,39 +3534,39 @@ P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\}$. The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer. The $i$-th neuron at layer $l$ recieves the result -$\vec{x}_j^{(l-1),\text{hidden} }$ from the $j$-th neuron at layer +$\hat{x}_j^{(l-1),\mathrm{hidden} }$ from the $j$-th neuron at layer $l-1$. The $i$-th neuron at layer $l$ weights all of the elements in -$\vec{x}_j^{(l-1),\text{hidden} }$ with a weight vector $\vec -w_{i,j}^{(l), \ \text{hidden} }$ with as many weigths as there are -elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias -$b_i^{(l), \ \text{hidden} }$: +$\hat{x}_j^{(l-1),\mathrm{hidden} }$ with a weight vector $\vec +w_{i,j}^{(l), \ \mathrm{hidden} }$ with as many weigths as there are +elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias +$b_i^{(l), \ \mathrm{hidden} }$: !bt \begin{aligned} -z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\ +z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 \\ -\vec{x}_j^{(l-1),\text{hidden} } +\hat{x}_j^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} !et -The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\vec{z}_{i}^{(l),\ \text{hidden}}$: +The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\hat{z}_{i}^{(l),\ \mathrm{hidden}}$: !bt \begin{aligned} -\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\ +\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\ &= \begin{pmatrix} -b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T \end{pmatrix} \begin{pmatrix} 1 & 1 & \dots & 1 \\ -\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } +\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \end{pmatrix} \end{aligned} !et