more boring typos
This commit is contained in:
mhjensen
@@ -387,7 +387,7 @@ depth respectively).
|
||||
The neurons in a layer will
|
||||
only be connected to a small region of the layer before it, instead of
|
||||
all of the neurons in a fully-connected manner. Moreover, the final
|
||||
output layer could for this specific image have dimensions \( 1\times 1 times 10 \),
|
||||
output layer could for this specific image have dimensions \( 1\times 1 \times 10 \),
|
||||
because by the
|
||||
end of the CNN architecture we will reduce the full image into a
|
||||
single vector of class scores, arranged along the depth
|
||||
|
||||
@@ -381,17 +381,17 @@ and biases for each layer.
|
||||
|
||||
<p>
|
||||
It is assumed that there are no weights and
|
||||
bias at the input layer, so \( P = \{ P_{\text{hidden}},
|
||||
P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} }
|
||||
bias at the input layer, so \( P = \{ P_{\mathrm{hidden}},
|
||||
P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} }
|
||||
\times 2 \) matrix.
|
||||
|
||||
<p>
|
||||
The first column in \( P_{\text{hidden} } \) represents
|
||||
The first column in \( P_{\mathrm{hidden} } \) represents
|
||||
the bias for each neuron in the hidden layer and the second column
|
||||
represents the weigths for each neuron. If there are \( N_{\text{output}
|
||||
} \) neurons in the output layer, then \( P_{\text{output}} \) is a
|
||||
\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first
|
||||
represents the weigths for each neuron. If there are \( N_{\mathrm{output}
|
||||
} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a
|
||||
\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first
|
||||
column represents the bias of each neuron and the remaining columns
|
||||
represents the weights to each neuron.
|
||||
|
||||
|
||||
@@ -381,7 +381,7 @@ $$
|
||||
or, in terms of weights and biases for each layer:
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -405,10 +405,10 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \)
|
||||
or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \)
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
|
||||
$$
|
||||
|
||||
<p>
|
||||
|
||||
@@ -380,8 +380,8 @@ extension of it into a deep neural network would (hopefully) be
|
||||
painless.
|
||||
|
||||
<p>
|
||||
For simplicity, we assume that the input is an array \( \vec x =
|
||||
(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
For simplicity, we assume that the input is an array
|
||||
\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
network should find \( P \) such that it fulfills <a href="._NeuralNet-bs092.html#mjx-eqn-19">(19)</a>.
|
||||
|
||||
<p>
|
||||
|
||||
@@ -366,24 +366,24 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec93" class="anchor">Feedforward </h2>
|
||||
|
||||
<p>
|
||||
First, a feedforward of the inputs must be done. This means that \( \vec
|
||||
x \) must be passed through an input layer, a hidden layer and a output
|
||||
First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
|
||||
must be passed through an input layer, a hidden layer and a output
|
||||
layer. The input layer in this case, does not need to process the
|
||||
data any further. The input layer will consist of \( N_{\text{input} } \)
|
||||
data any further. The input layer will consist of \( N_{\mathrm{input} } \)
|
||||
neurons, passing its element to each neuron in the hidden layer. The
|
||||
number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
|
||||
number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
|
||||
|
||||
<p>
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \)
|
||||
and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
|
||||
and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
at the input layer is:
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
|
||||
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
|
||||
@@ -369,28 +369,28 @@ MathJax.Hub.Config({
|
||||
The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector:
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
|
||||
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
x_1 & x_2 & \dots & x_N
|
||||
\end{pmatrix} \\
|
||||
&= \vec{p}_{i, \text{hidden}}^T X
|
||||
&= \hat{p}_{i, \mathrm{hidden}}^T X
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row
|
||||
in \( P_{\text{hidden} } \), which contains the weights for the neural
|
||||
It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
|
||||
in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
|
||||
network to minimize according to <a href="._NeuralNet-bs092.html#mjx-eqn-19">(19)</a>.
|
||||
|
||||
<p>
|
||||
After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \)
|
||||
After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
|
||||
in the hidden layer, the vector will be sent to an activation function
|
||||
\( a_i(\vec{z}) \). In this example, the sigmoid function has been used:
|
||||
\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
|
||||
|
||||
$$
|
||||
f(z) = \frac{1}{1 + \exp{(-z)}}.
|
||||
|
||||
@@ -366,20 +366,20 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec95" class="anchor">Output </h2>
|
||||
|
||||
<p>
|
||||
The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is:
|
||||
The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
|
||||
|
||||
$$
|
||||
\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
|
||||
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
|
||||
$$
|
||||
|
||||
<p>
|
||||
The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer.
|
||||
The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
|
||||
|
||||
<p>
|
||||
The output layer consist of one neuron in this case, and combines the
|
||||
output from each of the neurons in the hidden layers. The output layer
|
||||
combines the results from the hidden layer using some weights \(
|
||||
w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case,
|
||||
w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
|
||||
it is assumes that the number of neurons in the output layer is one.
|
||||
|
||||
<p>
|
||||
@@ -389,36 +389,36 @@ layer described previously.
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{1,j}^{\text{output}} & =
|
||||
z_{1,j}^{\mathrm{output}} & =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{\text{hidden}}
|
||||
\hat{x}_j^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
|
||||
$$
|
||||
\vec{z}_{1}^{\text{output}} =
|
||||
\hat{z}_{1}^{\mathrm{output}} =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
|
||||
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
<p>
|
||||
In this case we seek a continous range of values since we are
|
||||
approximating a function. This means that after computing
|
||||
\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final
|
||||
\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
|
||||
output of the network.
|
||||
|
||||
<p>
|
||||
|
||||
@@ -375,12 +375,12 @@ function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \),
|
||||
goes as follows:
|
||||
|
||||
$$
|
||||
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
$$
|
||||
|
||||
<p>
|
||||
for a number of iterations or until $ \big|\big| \vec
|
||||
\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
given tolerance.
|
||||
|
||||
<p>
|
||||
|
||||
@@ -368,17 +368,17 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
In our case, we have to minimize the cost function \( c(x, P) \) with
|
||||
respect to the two sets of weights and bisases, that is for the hidden
|
||||
layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output}
|
||||
layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output}
|
||||
} \) .
|
||||
|
||||
<p>
|
||||
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is
|
||||
This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
|
||||
updated by
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
|
||||
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
|
||||
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
|
||||
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
|
||||
@@ -368,13 +368,13 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
As previously stated, a Deep Neural Network (DNN) follows the same
|
||||
concept of a neural network, but having more than one hidden
|
||||
layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The
|
||||
layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The
|
||||
input is still assumed to be an array of size \( 1 \times N \). The
|
||||
network must now try to optimalize its output with respect to the
|
||||
collection of weigths and biases \( P = \big\{P_{\text{input} }, \
|
||||
P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \).
|
||||
collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \
|
||||
P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
|
||||
@@ -370,40 +370,40 @@ The feedforward step is similar to as for the neural netowork, but now consideri
|
||||
|
||||
<p>
|
||||
The \( i \)-th neuron at layer \( l \) recieves the result
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are
|
||||
elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \text{hidden} } \):
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are
|
||||
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \mathrm{hidden} } \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
|
||||
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{(l-1),\text{hidden} }
|
||||
\hat{x}_j^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \):
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
|
||||
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
|
||||
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
@@ -3172,7 +3172,7 @@ depth respectively).
|
||||
The neurons in a layer will
|
||||
only be connected to a small region of the layer before it, instead of
|
||||
all of the neurons in a fully-connected manner. Moreover, the final
|
||||
output layer could for this specific image have dimensions \( 1\times 1 times 10 \),
|
||||
output layer could for this specific image have dimensions \( 1\times 1 \times 10 \),
|
||||
because by the
|
||||
end of the CNN architecture we will reduce the full image into a
|
||||
single vector of class scores, arranged along the depth
|
||||
@@ -3821,17 +3821,17 @@ and biases for each layer.
|
||||
|
||||
<p>
|
||||
It is assumed that there are no weights and
|
||||
bias at the input layer, so \( P = \{ P_{\text{hidden}},
|
||||
P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} }
|
||||
bias at the input layer, so \( P = \{ P_{\mathrm{hidden}},
|
||||
P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} }
|
||||
\times 2 \) matrix.
|
||||
|
||||
<p>
|
||||
The first column in \( P_{\text{hidden} } \) represents
|
||||
The first column in \( P_{\mathrm{hidden} } \) represents
|
||||
the bias for each neuron in the hidden layer and the second column
|
||||
represents the weigths for each neuron. If there are \( N_{\text{output}
|
||||
} \) neurons in the output layer, then \( P_{\text{output}} \) is a
|
||||
\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first
|
||||
represents the weigths for each neuron. If there are \( N_{\mathrm{output}
|
||||
} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a
|
||||
\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first
|
||||
column represents the bias of each neuron and the remaining columns
|
||||
represents the weights to each neuron.
|
||||
</section>
|
||||
@@ -3914,7 +3914,7 @@ or, in terms of weights and biases for each layer:
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -3943,11 +3943,11 @@ $$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \)
|
||||
or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \)
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
|
||||
$$
|
||||
<p> <br>
|
||||
</section>
|
||||
@@ -3971,8 +3971,8 @@ extension of it into a deep neural network would (hopefully) be
|
||||
painless.
|
||||
|
||||
<p>
|
||||
For simplicity, we assume that the input is an array \( \vec x =
|
||||
(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
For simplicity, we assume that the input is an array
|
||||
\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
network should find \( P \) such that it fulfills <a href="#mjx-eqn-19">(19)</a>.
|
||||
</section>
|
||||
|
||||
@@ -3981,25 +3981,25 @@ network should find \( P \) such that it fulfills <a href="#mjx-eqn-19">(19)</a>
|
||||
<h2 id="___sec93">Feedforward </h2>
|
||||
|
||||
<p>
|
||||
First, a feedforward of the inputs must be done. This means that \( \vec
|
||||
x \) must be passed through an input layer, a hidden layer and a output
|
||||
First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
|
||||
must be passed through an input layer, a hidden layer and a output
|
||||
layer. The input layer in this case, does not need to process the
|
||||
data any further. The input layer will consist of \( N_{\text{input} } \)
|
||||
data any further. The input layer will consist of \( N_{\mathrm{input} } \)
|
||||
neurons, passing its element to each neuron in the hidden layer. The
|
||||
number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
|
||||
number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
|
||||
|
||||
<p>
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \)
|
||||
and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
|
||||
and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
at the input layer is:
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
|
||||
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
@@ -4019,29 +4019,29 @@ The result after weighting the input at the \( i \)-th hidden neuron can be writ
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
|
||||
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
x_1 & x_2 & \dots & x_N
|
||||
\end{pmatrix} \\
|
||||
&= \vec{p}_{i, \text{hidden}}^T X
|
||||
&= \hat{p}_{i, \mathrm{hidden}}^T X
|
||||
\end{aligned}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row
|
||||
in \( P_{\text{hidden} } \), which contains the weights for the neural
|
||||
It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
|
||||
in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
|
||||
network to minimize according to <a href="#mjx-eqn-19">(19)</a>.
|
||||
|
||||
<p>
|
||||
After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \)
|
||||
After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
|
||||
in the hidden layer, the vector will be sent to an activation function
|
||||
\( a_i(\vec{z}) \). In this example, the sigmoid function has been used:
|
||||
\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
@@ -4055,22 +4055,22 @@ $$
|
||||
<h2 id="___sec95">Output </h2>
|
||||
|
||||
<p>
|
||||
The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is:
|
||||
The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
|
||||
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer.
|
||||
The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
|
||||
|
||||
<p>
|
||||
The output layer consist of one neuron in this case, and combines the
|
||||
output from each of the neurons in the hidden layers. The output layer
|
||||
combines the results from the hidden layer using some weights \(
|
||||
w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case,
|
||||
w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
|
||||
it is assumes that the number of neurons in the output layer is one.
|
||||
|
||||
<p>
|
||||
@@ -4081,30 +4081,30 @@ layer described previously.
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{1,j}^{\text{output}} & =
|
||||
z_{1,j}^{\mathrm{output}} & =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{\text{hidden}}
|
||||
\hat{x}_j^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\vec{z}_{1}^{\text{output}} =
|
||||
\hat{z}_{1}^{\mathrm{output}} =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
|
||||
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
<p> <br>
|
||||
@@ -4112,8 +4112,8 @@ $$
|
||||
<p>
|
||||
In this case we seek a continous range of values since we are
|
||||
approximating a function. This means that after computing
|
||||
\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final
|
||||
\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
|
||||
output of the network.
|
||||
</section>
|
||||
|
||||
@@ -4244,13 +4244,13 @@ goes as follows:
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
for a number of iterations or until $ \big|\big| \vec
|
||||
\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
given tolerance.
|
||||
|
||||
<p>
|
||||
@@ -4267,18 +4267,18 @@ the elements in \( \vec \omega \).
|
||||
<p>
|
||||
In our case, we have to minimize the cost function \( c(x, P) \) with
|
||||
respect to the two sets of weights and bisases, that is for the hidden
|
||||
layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output}
|
||||
layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output}
|
||||
} \) .
|
||||
|
||||
<p>
|
||||
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is
|
||||
This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
|
||||
updated by
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{aligned}
|
||||
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
|
||||
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
|
||||
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
|
||||
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
|
||||
\end{aligned}
|
||||
$$
|
||||
<p> <br>
|
||||
@@ -4333,13 +4333,13 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
|
||||
<p>
|
||||
As previously stated, a Deep Neural Network (DNN) follows the same
|
||||
concept of a neural network, but having more than one hidden
|
||||
layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The
|
||||
layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The
|
||||
input is still assumed to be an array of size \( 1 \times N \). The
|
||||
network must now try to optimalize its output with respect to the
|
||||
collection of weigths and biases \( P = \big\{P_{\text{input} }, \
|
||||
P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \).
|
||||
collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \
|
||||
P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
|
||||
</section>
|
||||
|
||||
|
||||
@@ -4351,43 +4351,43 @@ The feedforward step is similar to as for the neural netowork, but now consideri
|
||||
|
||||
<p>
|
||||
The \( i \)-th neuron at layer \( l \) recieves the result
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are
|
||||
elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \text{hidden} } \):
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are
|
||||
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \mathrm{hidden} } \):
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
|
||||
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{(l-1),\text{hidden} }
|
||||
\hat{x}_j^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \):
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
|
||||
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
|
||||
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
@@ -3072,7 +3072,7 @@ depth respectively).
|
||||
The neurons in a layer will
|
||||
only be connected to a small region of the layer before it, instead of
|
||||
all of the neurons in a fully-connected manner. Moreover, the final
|
||||
output layer could for this specific image have dimensions \( 1\times 1 times 10 \),
|
||||
output layer could for this specific image have dimensions \( 1\times 1 \times 10 \),
|
||||
because by the
|
||||
end of the CNN architecture we will reduce the full image into a
|
||||
single vector of class scores, arranged along the depth
|
||||
@@ -3703,17 +3703,17 @@ and biases for each layer.
|
||||
|
||||
<p>
|
||||
It is assumed that there are no weights and
|
||||
bias at the input layer, so \( P = \{ P_{\text{hidden}},
|
||||
P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} }
|
||||
bias at the input layer, so \( P = \{ P_{\mathrm{hidden}},
|
||||
P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} }
|
||||
\times 2 \) matrix.
|
||||
|
||||
<p>
|
||||
The first column in \( P_{\text{hidden} } \) represents
|
||||
The first column in \( P_{\mathrm{hidden} } \) represents
|
||||
the bias for each neuron in the hidden layer and the second column
|
||||
represents the weigths for each neuron. If there are \( N_{\text{output}
|
||||
} \) neurons in the output layer, then \( P_{\text{output}} \) is a
|
||||
\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first
|
||||
represents the weigths for each neuron. If there are \( N_{\mathrm{output}
|
||||
} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a
|
||||
\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first
|
||||
column represents the bias of each neuron and the remaining columns
|
||||
represents the weights to each neuron.
|
||||
|
||||
@@ -3789,7 +3789,7 @@ $$
|
||||
or, in terms of weights and biases for each layer:
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -3813,10 +3813,10 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \)
|
||||
or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \)
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -3839,8 +3839,8 @@ extension of it into a deep neural network would (hopefully) be
|
||||
painless.
|
||||
|
||||
<p>
|
||||
For simplicity, we assume that the input is an array \( \vec x =
|
||||
(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
For simplicity, we assume that the input is an array
|
||||
\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
network should find \( P \) such that it fulfills \eqref{eq:min}.
|
||||
|
||||
<p>
|
||||
@@ -3849,24 +3849,24 @@ network should find \( P \) such that it fulfills \eqref{eq:min}.
|
||||
<h2 id="___sec93">Feedforward </h2>
|
||||
|
||||
<p>
|
||||
First, a feedforward of the inputs must be done. This means that \( \vec
|
||||
x \) must be passed through an input layer, a hidden layer and a output
|
||||
First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
|
||||
must be passed through an input layer, a hidden layer and a output
|
||||
layer. The input layer in this case, does not need to process the
|
||||
data any further. The input layer will consist of \( N_{\text{input} } \)
|
||||
data any further. The input layer will consist of \( N_{\mathrm{input} } \)
|
||||
neurons, passing its element to each neuron in the hidden layer. The
|
||||
number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
|
||||
number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
|
||||
|
||||
<p>
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \)
|
||||
and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
|
||||
and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
at the input layer is:
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
|
||||
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
@@ -3884,28 +3884,28 @@ $$
|
||||
The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector:
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
|
||||
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
x_1 & x_2 & \dots & x_N
|
||||
\end{pmatrix} \\
|
||||
&= \vec{p}_{i, \text{hidden}}^T X
|
||||
&= \hat{p}_{i, \mathrm{hidden}}^T X
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row
|
||||
in \( P_{\text{hidden} } \), which contains the weights for the neural
|
||||
It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
|
||||
in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
|
||||
network to minimize according to \eqref{eq:min}.
|
||||
|
||||
<p>
|
||||
After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \)
|
||||
After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
|
||||
in the hidden layer, the vector will be sent to an activation function
|
||||
\( a_i(\vec{z}) \). In this example, the sigmoid function has been used:
|
||||
\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
|
||||
|
||||
$$
|
||||
f(z) = \frac{1}{1 + \exp{(-z)}}.
|
||||
@@ -3917,20 +3917,20 @@ $$
|
||||
<h2 id="___sec95">Output </h2>
|
||||
|
||||
<p>
|
||||
The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is:
|
||||
The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
|
||||
|
||||
$$
|
||||
\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
|
||||
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
|
||||
$$
|
||||
|
||||
<p>
|
||||
The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer.
|
||||
The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
|
||||
|
||||
<p>
|
||||
The output layer consist of one neuron in this case, and combines the
|
||||
output from each of the neurons in the hidden layers. The output layer
|
||||
combines the results from the hidden layer using some weights \(
|
||||
w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case,
|
||||
w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
|
||||
it is assumes that the number of neurons in the output layer is one.
|
||||
|
||||
<p>
|
||||
@@ -3940,36 +3940,36 @@ layer described previously.
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{1,j}^{\text{output}} & =
|
||||
z_{1,j}^{\mathrm{output}} & =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{\text{hidden}}
|
||||
\hat{x}_j^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
|
||||
$$
|
||||
\vec{z}_{1}^{\text{output}} =
|
||||
\hat{z}_{1}^{\mathrm{output}} =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
|
||||
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
<p>
|
||||
In this case we seek a continous range of values since we are
|
||||
approximating a function. This means that after computing
|
||||
\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final
|
||||
\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
|
||||
output of the network.
|
||||
|
||||
<p>
|
||||
@@ -4095,12 +4095,12 @@ function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \),
|
||||
goes as follows:
|
||||
|
||||
$$
|
||||
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
$$
|
||||
|
||||
<p>
|
||||
for a number of iterations or until $ \big|\big| \vec
|
||||
\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
given tolerance.
|
||||
|
||||
<p>
|
||||
@@ -4117,17 +4117,17 @@ the elements in \( \vec \omega \).
|
||||
<p>
|
||||
In our case, we have to minimize the cost function \( c(x, P) \) with
|
||||
respect to the two sets of weights and bisases, that is for the hidden
|
||||
layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output}
|
||||
layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output}
|
||||
} \) .
|
||||
|
||||
<p>
|
||||
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is
|
||||
This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
|
||||
updated by
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
|
||||
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
|
||||
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
|
||||
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
@@ -4180,13 +4180,13 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
|
||||
<p>
|
||||
As previously stated, a Deep Neural Network (DNN) follows the same
|
||||
concept of a neural network, but having more than one hidden
|
||||
layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The
|
||||
layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The
|
||||
input is still assumed to be an array of size \( 1 \times N \). The
|
||||
network must now try to optimalize its output with respect to the
|
||||
collection of weigths and biases \( P = \big\{P_{\text{input} }, \
|
||||
P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \).
|
||||
collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \
|
||||
P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -4198,40 +4198,40 @@ The feedforward step is similar to as for the neural netowork, but now consideri
|
||||
|
||||
<p>
|
||||
The \( i \)-th neuron at layer \( l \) recieves the result
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are
|
||||
elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \text{hidden} } \):
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are
|
||||
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \mathrm{hidden} } \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
|
||||
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{(l-1),\text{hidden} }
|
||||
\hat{x}_j^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \):
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
|
||||
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
|
||||
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
@@ -3077,7 +3077,7 @@ depth respectively).
|
||||
The neurons in a layer will
|
||||
only be connected to a small region of the layer before it, instead of
|
||||
all of the neurons in a fully-connected manner. Moreover, the final
|
||||
output layer could for this specific image have dimensions \( 1\times 1 times 10 \),
|
||||
output layer could for this specific image have dimensions \( 1\times 1 \times 10 \),
|
||||
because by the
|
||||
end of the CNN architecture we will reduce the full image into a
|
||||
single vector of class scores, arranged along the depth
|
||||
@@ -3708,17 +3708,17 @@ and biases for each layer.
|
||||
|
||||
<p>
|
||||
It is assumed that there are no weights and
|
||||
bias at the input layer, so \( P = \{ P_{\text{hidden}},
|
||||
P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} }
|
||||
bias at the input layer, so \( P = \{ P_{\mathrm{hidden}},
|
||||
P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} }
|
||||
\times 2 \) matrix.
|
||||
|
||||
<p>
|
||||
The first column in \( P_{\text{hidden} } \) represents
|
||||
The first column in \( P_{\mathrm{hidden} } \) represents
|
||||
the bias for each neuron in the hidden layer and the second column
|
||||
represents the weigths for each neuron. If there are \( N_{\text{output}
|
||||
} \) neurons in the output layer, then \( P_{\text{output}} \) is a
|
||||
\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first
|
||||
represents the weigths for each neuron. If there are \( N_{\mathrm{output}
|
||||
} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a
|
||||
\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first
|
||||
column represents the bias of each neuron and the remaining columns
|
||||
represents the weights to each neuron.
|
||||
|
||||
@@ -3794,7 +3794,7 @@ $$
|
||||
or, in terms of weights and biases for each layer:
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -3818,10 +3818,10 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \)
|
||||
or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \)
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -3844,8 +3844,8 @@ extension of it into a deep neural network would (hopefully) be
|
||||
painless.
|
||||
|
||||
<p>
|
||||
For simplicity, we assume that the input is an array \( \vec x =
|
||||
(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
For simplicity, we assume that the input is an array
|
||||
\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
network should find \( P \) such that it fulfills \eqref{eq:min}.
|
||||
|
||||
<p>
|
||||
@@ -3854,24 +3854,24 @@ network should find \( P \) such that it fulfills \eqref{eq:min}.
|
||||
<h2 id="___sec93">Feedforward </h2>
|
||||
|
||||
<p>
|
||||
First, a feedforward of the inputs must be done. This means that \( \vec
|
||||
x \) must be passed through an input layer, a hidden layer and a output
|
||||
First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
|
||||
must be passed through an input layer, a hidden layer and a output
|
||||
layer. The input layer in this case, does not need to process the
|
||||
data any further. The input layer will consist of \( N_{\text{input} } \)
|
||||
data any further. The input layer will consist of \( N_{\mathrm{input} } \)
|
||||
neurons, passing its element to each neuron in the hidden layer. The
|
||||
number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
|
||||
number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
|
||||
|
||||
<p>
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \)
|
||||
and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
|
||||
and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
at the input layer is:
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
|
||||
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
@@ -3889,28 +3889,28 @@ $$
|
||||
The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector:
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
|
||||
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
x_1 & x_2 & \dots & x_N
|
||||
\end{pmatrix} \\
|
||||
&= \vec{p}_{i, \text{hidden}}^T X
|
||||
&= \hat{p}_{i, \mathrm{hidden}}^T X
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row
|
||||
in \( P_{\text{hidden} } \), which contains the weights for the neural
|
||||
It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
|
||||
in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
|
||||
network to minimize according to \eqref{eq:min}.
|
||||
|
||||
<p>
|
||||
After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \)
|
||||
After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
|
||||
in the hidden layer, the vector will be sent to an activation function
|
||||
\( a_i(\vec{z}) \). In this example, the sigmoid function has been used:
|
||||
\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
|
||||
|
||||
$$
|
||||
f(z) = \frac{1}{1 + \exp{(-z)}}.
|
||||
@@ -3922,20 +3922,20 @@ $$
|
||||
<h2 id="___sec95">Output </h2>
|
||||
|
||||
<p>
|
||||
The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is:
|
||||
The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
|
||||
|
||||
$$
|
||||
\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
|
||||
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
|
||||
$$
|
||||
|
||||
<p>
|
||||
The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer.
|
||||
The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
|
||||
|
||||
<p>
|
||||
The output layer consist of one neuron in this case, and combines the
|
||||
output from each of the neurons in the hidden layers. The output layer
|
||||
combines the results from the hidden layer using some weights \(
|
||||
w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case,
|
||||
w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
|
||||
it is assumes that the number of neurons in the output layer is one.
|
||||
|
||||
<p>
|
||||
@@ -3945,36 +3945,36 @@ layer described previously.
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{1,j}^{\text{output}} & =
|
||||
z_{1,j}^{\mathrm{output}} & =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{\text{hidden}}
|
||||
\hat{x}_j^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
|
||||
$$
|
||||
\vec{z}_{1}^{\text{output}} =
|
||||
\hat{z}_{1}^{\mathrm{output}} =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
|
||||
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
<p>
|
||||
In this case we seek a continous range of values since we are
|
||||
approximating a function. This means that after computing
|
||||
\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final
|
||||
\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
|
||||
output of the network.
|
||||
|
||||
<p>
|
||||
@@ -4100,12 +4100,12 @@ function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \),
|
||||
goes as follows:
|
||||
|
||||
$$
|
||||
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
$$
|
||||
|
||||
<p>
|
||||
for a number of iterations or until $ \big|\big| \vec
|
||||
\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
given tolerance.
|
||||
|
||||
<p>
|
||||
@@ -4122,17 +4122,17 @@ the elements in \( \vec \omega \).
|
||||
<p>
|
||||
In our case, we have to minimize the cost function \( c(x, P) \) with
|
||||
respect to the two sets of weights and bisases, that is for the hidden
|
||||
layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output}
|
||||
layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output}
|
||||
} \) .
|
||||
|
||||
<p>
|
||||
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is
|
||||
This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
|
||||
updated by
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
|
||||
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
|
||||
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
|
||||
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
@@ -4185,13 +4185,13 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
|
||||
<p>
|
||||
As previously stated, a Deep Neural Network (DNN) follows the same
|
||||
concept of a neural network, but having more than one hidden
|
||||
layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The
|
||||
layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The
|
||||
input is still assumed to be an array of size \( 1 \times N \). The
|
||||
network must now try to optimalize its output with respect to the
|
||||
collection of weigths and biases \( P = \big\{P_{\text{input} }, \
|
||||
P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \).
|
||||
collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \
|
||||
P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -4203,40 +4203,40 @@ The feedforward step is similar to as for the neural netowork, but now consideri
|
||||
|
||||
<p>
|
||||
The \( i \)-th neuron at layer \( l \) recieves the result
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are
|
||||
elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \text{hidden} } \):
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are
|
||||
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \mathrm{hidden} } \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
|
||||
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{(l-1),\text{hidden} }
|
||||
\hat{x}_j^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \):
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
|
||||
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
|
||||
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
@@ -3378,7 +3378,7 @@
|
||||
"The neurons in a layer will\n",
|
||||
"only be connected to a small region of the layer before it, instead of\n",
|
||||
"all of the neurons in a fully-connected manner. Moreover, the final\n",
|
||||
"output layer could for this specific image have dimensions $1\\times 1 times 10$, \n",
|
||||
"output layer could for this specific image have dimensions $1\\times 1 \\times 10$, \n",
|
||||
"because by the\n",
|
||||
"end of the CNN architecture we will reduce the full image into a\n",
|
||||
"single vector of class scores, arranged along the depth\n",
|
||||
@@ -4071,16 +4071,16 @@
|
||||
"and biases for each layer. \n",
|
||||
"\n",
|
||||
"It is assumed that there are no weights and\n",
|
||||
"bias at the input layer, so $P = \\{ P_{\\text{hidden}},\n",
|
||||
"P_{\\text{output}} \\}$. If there are $N_{\\text{hidden} }$ neurons in\n",
|
||||
"the hidden layer, then $P_{\\text{hidden}}$ is an $N_{\\text{hidden} }\n",
|
||||
"bias at the input layer, so $P = \\{ P_{\\mathrm{hidden}},\n",
|
||||
"P_{\\mathrm{output}} \\}$. If there are $N_{\\mathrm{hidden} }$ neurons in\n",
|
||||
"the hidden layer, then $P_{\\mathrm{hidden}}$ is an $N_{\\mathrm{hidden} }\n",
|
||||
"\\times 2$ matrix. \n",
|
||||
"\n",
|
||||
"The first column in $P_{\\text{hidden} }$ represents\n",
|
||||
"The first column in $P_{\\mathrm{hidden} }$ represents\n",
|
||||
"the bias for each neuron in the hidden layer and the second column\n",
|
||||
"represents the weigths for each neuron. If there are $N_{\\text{output}\n",
|
||||
"}$ neurons in the output layer, then $P_{\\text{output}} $ is a\n",
|
||||
"$N_{\\text{output} } \\times (1 + N_{\\text{hidden} })$ matrix. Its first\n",
|
||||
"represents the weigths for each neuron. If there are $N_{\\mathrm{output}\n",
|
||||
"}$ neurons in the output layer, then $P_{\\mathrm{output}} $ is a\n",
|
||||
"$N_{\\mathrm{output} } \\times (1 + N_{\\mathrm{hidden} })$ matrix. Its first\n",
|
||||
"column represents the bias of each neuron and the remaining columns\n",
|
||||
"represents the weights to each neuron.\n",
|
||||
"\n",
|
||||
@@ -4183,7 +4183,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) \\big)^2 \\Big\\}\n",
|
||||
"\\min_{P_{\\mathrm{hidden} }, \\ P_{\\mathrm{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\mathrm{hidden} }, P_{\\mathrm{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\mathrm{hidden} }, P_{\\mathrm{output} }\\}) \\big)^2 \\Big\\}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -4232,7 +4232,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"or in terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$"
|
||||
"or in terms of $P_{\\mathrm{hidden} }$ and $P_{\\mathrm{output} }$"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -4240,7 +4240,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(x, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n",
|
||||
"\\min_{P_{\\mathrm{hidden} }, \\ P_{\\mathrm{output} }} c(x, \\{P_{\\mathrm{hidden} }, P_{\\mathrm{output} }\\})\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -4263,21 +4263,21 @@
|
||||
"extension of it into a deep neural network would (hopefully) be\n",
|
||||
"painless.\n",
|
||||
"\n",
|
||||
"For simplicity, we assume that the input is an array $\\vec x =\n",
|
||||
"(x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural\n",
|
||||
"For simplicity, we assume that the input is an array \n",
|
||||
"$\\hat{x}= (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural\n",
|
||||
"network should find $P$ such that it fulfills ([eq:min](#eq:min)).\n",
|
||||
"\n",
|
||||
"## Feedforward\n",
|
||||
"\n",
|
||||
"First, a feedforward of the inputs must be done. This means that $\\vec\n",
|
||||
" x$ must be passed through an input layer, a hidden layer and a output\n",
|
||||
"First, a feedforward of the inputs must be done. This means that $\\hat{x}$ \n",
|
||||
"must be passed through an input layer, a hidden layer and a output\n",
|
||||
" layer. The input layer in this case, does not need to process the\n",
|
||||
" data any further. The input layer will consist of $N_{\\text{input} }$\n",
|
||||
" data any further. The input layer will consist of $N_{\\mathrm{input} }$\n",
|
||||
" neurons, passing its element to each neuron in the hidden layer. The\n",
|
||||
" number of neurons in the hidden layer will be $N_{\\text{hidden} }$.\n",
|
||||
" number of neurons in the hidden layer will be $N_{\\mathrm{hidden} }$.\n",
|
||||
"\n",
|
||||
"For the $i$-th in the hidden layer with weight $w_i^{\\text{hidden} }$\n",
|
||||
"and bias $b_i^{\\text{hidden} }$, the weighting from the $j$-th neuron\n",
|
||||
"For the $i$-th in the hidden layer with weight $w_i^{\\mathrm{hidden} }$\n",
|
||||
"and bias $b_i^{\\mathrm{hidden} }$, the weighting from the $j$-th neuron\n",
|
||||
"at the input layer is:"
|
||||
]
|
||||
},
|
||||
@@ -4287,10 +4287,10 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"z_{i,j}^{\\text{hidden}} &= b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_j \\\\\n",
|
||||
"z_{i,j}^{\\mathrm{hidden}} &= b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}}x_j \\\\\n",
|
||||
"&= \n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n",
|
||||
"b_i^{\\mathrm{hidden}} & w_i^{\\mathrm{hidden}}\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"1 \\\\\n",
|
||||
@@ -4316,16 +4316,16 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"\\vec{z}_{i}^{\\text{hidden}} &= \\Big( b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_1 , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_2, \\ \\dots \\, , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_N\\Big) \\\\\n",
|
||||
"\\hat{z}_{i}^{\\mathrm{hidden}} &= \\Big( b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}}x_1 , \\ b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}} x_2, \\ \\dots \\, , \\ b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}} x_N\\Big) \\\\\n",
|
||||
"&= \n",
|
||||
"\\begin{pmatrix}\n",
|
||||
" b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n",
|
||||
" b_i^{\\mathrm{hidden}} & w_i^{\\mathrm{hidden}}\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"1 & 1 & \\dots & 1 \\\\\n",
|
||||
"x_1 & x_2 & \\dots & x_N\n",
|
||||
"\\end{pmatrix} \\\\\n",
|
||||
"&= \\vec{p}_{i, \\text{hidden}}^T X\n",
|
||||
"&= \\hat{p}_{i, \\mathrm{hidden}}^T X\n",
|
||||
"\\end{aligned}\n",
|
||||
"$$"
|
||||
]
|
||||
@@ -4334,13 +4334,13 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"It is the vector $\\vec{p}_{i, \\text{hidden}}^T$ that defines each row\n",
|
||||
"in $P_{\\text{hidden} }$, which contains the weights for the neural\n",
|
||||
"It is the vector $\\hat{p}_{i, \\mathrm{hidden}}^T$ that defines each row\n",
|
||||
"in $P_{\\mathrm{hidden} }$, which contains the weights for the neural\n",
|
||||
"network to minimize according to ([eq:min](#eq:min)).\n",
|
||||
"\n",
|
||||
"After having found $\\vec{z}_{i}^{\\text{hidden}} $ for every neuron $i$\n",
|
||||
"After having found $\\hat{z}_{i}^{\\mathrm{hidden}} $ for every neuron $i$\n",
|
||||
"in the hidden layer, the vector will be sent to an activation function\n",
|
||||
"$a_i(\\vec{z})$. In this example, the sigmoid function has been used:\n",
|
||||
"$a_i(\\hat{z})$. In this example, the sigmoid function has been used:\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"f(z) = \\frac{1}{1 + \\exp{(-z)}}.\n",
|
||||
@@ -4349,18 +4349,18 @@
|
||||
"\n",
|
||||
"## Output\n",
|
||||
"\n",
|
||||
"The output $\\vec{x}_i^{\\text{hidden} }$from each $i$-th hidden neuron is:\n",
|
||||
"The output $\\hat{x}_i^{\\mathrm{hidden}}$from each $i$-th hidden neuron is:\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\vec{x}_i^{\\text{hidden} } = f\\big( \\vec{z}_{i}^{\\text{hidden}} \\big).\n",
|
||||
"\\hat{x}_i^{\\mathrm{hidden} } = f\\big( \\hat{z}_{i}^{\\mathrm{hidden}} \\big).\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"The outputs $\\vec{x}_i^{\\text{hidden} } $ are then sent to the output layer. \n",
|
||||
"The outputs $\\hat{x}_i^{\\mathrm{hidden} } $ are then sent to the output layer. \n",
|
||||
"\n",
|
||||
"The output layer consist of one neuron in this case, and combines the\n",
|
||||
"output from each of the neurons in the hidden layers. The output layer\n",
|
||||
"combines the results from the hidden layer using some weights $\n",
|
||||
"w_i^{\\text{output}}$ and biases $b_i^{\\text{output}}$. In this case,\n",
|
||||
"w_i^{\\mathrm{output}}$ and biases $b_i^{\\mathrm{output}}$. In this case,\n",
|
||||
"it is assumes that the number of neurons in the output layer is one.\n",
|
||||
"\n",
|
||||
"The procedure of weigthing the output neuron $j$ in the hidden layer\n",
|
||||
@@ -4374,13 +4374,13 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"z_{1,j}^{\\text{output}} & = \n",
|
||||
"z_{1,j}^{\\mathrm{output}} & = \n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"b_1^{\\text{output}} & \\vec{w}_1^{\\text{output}}\n",
|
||||
"b_1^{\\mathrm{output}} & \\hat{w}_1^{\\mathrm{output}}\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"1 \\\\\n",
|
||||
"\\vec{x}_j^{\\text{hidden}}\n",
|
||||
"\\hat{x}_j^{\\mathrm{hidden}}\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\end{aligned}\n",
|
||||
"$$"
|
||||
@@ -4390,23 +4390,23 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Expressing $z_{1,j}^{\\text{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:\n",
|
||||
"Expressing $z_{1,j}^{\\mathrm{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\vec{z}_{1}^{\\text{output}} = \n",
|
||||
"\\hat{z}_{1}^{\\mathrm{output}} = \n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"b_1^{\\text{output}} & \\vec{w}_1^{\\text{output}}\n",
|
||||
"b_1^{\\mathrm{output}} & \\hat{w}_1^{\\mathrm{output}}\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"1 & 1 & \\dots & 1 \\\\\n",
|
||||
"\\vec{x}_1^{\\text{hidden}} & \\vec{x}_2^{\\text{hidden}} & \\dots & \\vec{x}_N^{\\text{hidden}}\n",
|
||||
"\\hat{x}_1^{\\mathrm{hidden}} & \\hat{x}_2^{\\mathrm{hidden}} & \\dots & \\hat{x}_N^{\\mathrm{hidden}}\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"In this case we seek a continous range of values since we are\n",
|
||||
"approximating a function. This means that after computing\n",
|
||||
"$\\vec{z}_{1}^{\\text{output}}$ the neural network has finished its\n",
|
||||
"feedforward step, and $\\vec{z}_{1}^{\\text{output}}$ is the final\n",
|
||||
"$\\hat{z}_{1}^{\\mathrm{output}}$ the neural network has finished its\n",
|
||||
"feedforward step, and $\\hat{z}_{1}^{\\mathrm{output}}$ is the final\n",
|
||||
"output of the network.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -4555,7 +4555,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\vec \\omega_{\\text{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(x, \\vec \\omega),\n",
|
||||
"\\vec \\omega_{\\mathrm{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(x, \\vec \\omega),\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -4564,7 +4564,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"for a number of iterations or until $ \\big|\\big| \\vec\n",
|
||||
"\\omega_{\\text{new} } - \\vec \\omega \\big|\\big|$ is smaller than some\n",
|
||||
"\\omega_{\\mathrm{new} } - \\vec \\omega \\big|\\big|$ is smaller than some\n",
|
||||
"given tolerance.\n",
|
||||
"\n",
|
||||
"The value of $\\lambda$ decides how large steps the algorithm must take\n",
|
||||
@@ -4577,10 +4577,10 @@
|
||||
"\n",
|
||||
"In our case, we have to minimize the cost function $c(x, P)$ with\n",
|
||||
"respect to the two sets of weights and bisases, that is for the hidden\n",
|
||||
"layer $P_{\\text{hidden} }$ and for the ouput layer $P_{\\text{output}\n",
|
||||
"layer $P_{\\mathrm{hidden} }$ and for the ouput layer $P_{\\mathrm{output}\n",
|
||||
"}$ .\n",
|
||||
"\n",
|
||||
"This means that $P_{\\text{hidden} }$ and $P_{\\text{output} }$ is\n",
|
||||
"This means that $P_{\\mathrm{hidden} }$ and $P_{\\mathrm{output} }$ is\n",
|
||||
"updated by"
|
||||
]
|
||||
},
|
||||
@@ -4590,8 +4590,8 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(x, P) \\\\\n",
|
||||
"P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(x, P) \n",
|
||||
"P_{\\mathrm{hidden},\\mathrm{new}} &= P_{\\mathrm{hidden}} - \\lambda \\nabla_{P_{\\mathrm{hidden}}} c(x, P) \\\\\n",
|
||||
"P_{\\mathrm{output},\\mathrm{new}} &= P_{\\mathrm{output}} - \\lambda \\nabla_{P_{\\mathrm{output}}} c(x, P) \n",
|
||||
"\\end{aligned}\n",
|
||||
"$$"
|
||||
]
|
||||
@@ -4655,13 +4655,13 @@
|
||||
"\n",
|
||||
"As previously stated, a Deep Neural Network (DNN) follows the same\n",
|
||||
"concept of a neural network, but having more than one hidden\n",
|
||||
"layer. Suppose that the network has $N_{\\text{hidden}}$ hidden layers\n",
|
||||
"where the $l$-th layer has $N_{\\text{hidden}}^{(l)}$ neurons. The\n",
|
||||
"layer. Suppose that the network has $N_{\\mathrm{hidden}}$ hidden layers\n",
|
||||
"where the $l$-th layer has $N_{\\mathrm{hidden}}^{(l)}$ neurons. The\n",
|
||||
"input is still assumed to be an array of size $1 \\times N$. The\n",
|
||||
"network must now try to optimalize its output with respect to the\n",
|
||||
"collection of weigths and biases $P = \\big\\{P_{\\text{input} }, \\\n",
|
||||
"P_{\\text{hidden} }^{(1)}, \\ P_{\\text{hidden} }^{(2)}, \\ \\dots , \\\n",
|
||||
"P_{\\text{hidden} }^{(N_{\\text{hidden}})}, \\ P_{\\text{output} }\\big\\}$.\n",
|
||||
"collection of weigths and biases $P = \\big\\{P_{\\mathrm{input} }, \\\n",
|
||||
"P_{\\mathrm{hidden} }^{(1)}, \\ P_{\\mathrm{hidden} }^{(2)}, \\ \\dots , \\\n",
|
||||
"P_{\\mathrm{hidden} }^{(N_{\\mathrm{hidden}})}, \\ P_{\\mathrm{output} }\\big\\}$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Feed forward again\n",
|
||||
@@ -4669,12 +4669,12 @@
|
||||
"The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer. \n",
|
||||
"\n",
|
||||
"The $i$-th neuron at layer $l$ recieves the result\n",
|
||||
"$\\vec{x}_j^{(l-1),\\text{hidden} }$ from the $j$-th neuron at layer\n",
|
||||
"$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$ from the $j$-th neuron at layer\n",
|
||||
"$l-1$. The $i$-th neuron at layer $l$ weights all of the elements in\n",
|
||||
"$\\vec{x}_j^{(l-1),\\text{hidden} }$ with a weight vector $\\vec\n",
|
||||
"w_{i,j}^{(l), \\ \\text{hidden} }$ with as many weigths as there are\n",
|
||||
"elements in$\\vec{x}_j^{(l-1),\\text{hidden} }$, and adds a bias\n",
|
||||
"$b_i^{(l), \\ \\text{hidden} }$:"
|
||||
"$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$ with a weight vector $\\vec\n",
|
||||
"w_{i,j}^{(l), \\ \\mathrm{hidden} }$ with as many weigths as there are\n",
|
||||
"elements in$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$, and adds a bias\n",
|
||||
"$b_i^{(l), \\ \\mathrm{hidden} }$:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -4683,14 +4683,14 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"z_{i,j}^{(l),\\ \\text{hidden}} &= b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_j^{(l-1),\\text{hidden} } \\\\\n",
|
||||
"z_{i,j}^{(l),\\ \\mathrm{hidden}} &= b_i^{(l), \\ \\mathrm{hidden}} + \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\\hat{x}_j^{(l-1),\\mathrm{hidden} } \\\\\n",
|
||||
"&= \n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"b_i^{(l), \\ \\text{hidden}} & \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\n",
|
||||
"b_i^{(l), \\ \\mathrm{hidden}} & \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"1 \\\\\n",
|
||||
"\\vec{x}_j^{(l-1),\\text{hidden} }\n",
|
||||
"\\hat{x}_j^{(l-1),\\mathrm{hidden} }\n",
|
||||
"\\end{pmatrix} \n",
|
||||
"\\end{aligned}\n",
|
||||
"$$"
|
||||
@@ -4700,7 +4700,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\\vec{z}_{i}^{(l),\\ \\text{hidden}}$:"
|
||||
"The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\\hat{z}_{i}^{(l),\\ \\mathrm{hidden}}$:"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -4709,14 +4709,14 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"\\vec{z}_{i}^{(l),\\ \\text{hidden}} &= \\Big( b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_1^{(l-1),\\text{hidden} }, \\ \\dots \\ , \\ b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\text{hidden} } \\Big) \\\\\n",
|
||||
"\\hat{z}_{i}^{(l),\\ \\mathrm{hidden}} &= \\Big( b_i^{(l), \\ \\mathrm{hidden}} + \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\\hat{x}_1^{(l-1),\\mathrm{hidden} }, \\ \\dots \\ , \\ b_i^{(l), \\ \\mathrm{hidden}} + \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\mathrm{hidden} } \\Big) \\\\\n",
|
||||
"&= \n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"b_i^{(l), \\ \\text{hidden}} & \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\n",
|
||||
"b_i^{(l), \\ \\mathrm{hidden}} & \\big(\\hat{w}_{i}^{(l), \\ \\mathrm{hidden}}\\big)^T\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\begin{pmatrix}\n",
|
||||
"1 & 1 & \\dots & 1 \\\\\n",
|
||||
"\\vec{x}_{1}^{(l-1),\\text{hidden} } & \\vec{x}_{2}^{(l-1),\\text{hidden} } & \\dots & \\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\text{hidden} }\n",
|
||||
"\\hat{x}_{1}^{(l-1),\\mathrm{hidden} } & \\hat{x}_{2}^{(l-1),\\mathrm{hidden} } & \\dots & \\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\mathrm{hidden} }\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"\\end{aligned}\n",
|
||||
"$$"
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -2517,7 +2517,7 @@ depth respectively).
|
||||
The neurons in a layer will
|
||||
only be connected to a small region of the layer before it, instead of
|
||||
all of the neurons in a fully-connected manner. Moreover, the final
|
||||
output layer could for this specific image have dimensions $1\times 1 times 10$,
|
||||
output layer could for this specific image have dimensions $1\times 1 \times 10$,
|
||||
because by the
|
||||
end of the CNN architecture we will reduce the full image into a
|
||||
single vector of class scores, arranged along the depth
|
||||
@@ -3088,16 +3088,16 @@ neural network $N(x,P)$ with $P $ being the collection of the weights
|
||||
and biases for each layer.
|
||||
|
||||
It is assumed that there are no weights and
|
||||
bias at the input layer, so $P = \{ P_{\text{hidden}},
|
||||
P_{\text{output}} \}$. If there are $N_{\text{hidden} }$ neurons in
|
||||
the hidden layer, then $P_{\text{hidden}}$ is an $N_{\text{hidden} }
|
||||
bias at the input layer, so $P = \{ P_{\mathrm{hidden}},
|
||||
P_{\mathrm{output}} \}$. If there are $N_{\mathrm{hidden} }$ neurons in
|
||||
the hidden layer, then $P_{\mathrm{hidden}}$ is an $N_{\mathrm{hidden} }
|
||||
\times 2$ matrix.
|
||||
|
||||
The first column in $P_{\text{hidden} }$ represents
|
||||
The first column in $P_{\mathrm{hidden} }$ represents
|
||||
the bias for each neuron in the hidden layer and the second column
|
||||
represents the weigths for each neuron. If there are $N_{\text{output}
|
||||
}$ neurons in the output layer, then $P_{\text{output}} $ is a
|
||||
$N_{\text{output} } \times (1 + N_{\text{hidden} })$ matrix. Its first
|
||||
represents the weigths for each neuron. If there are $N_{\mathrm{output}
|
||||
}$ neurons in the output layer, then $P_{\mathrm{output}} $ is a
|
||||
$N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} })$ matrix. Its first
|
||||
column represents the bias of each neuron and the remaining columns
|
||||
represents the weights to each neuron.
|
||||
|
||||
@@ -3163,7 +3163,7 @@ or, in terms of weights and biases for each layer:
|
||||
|
||||
!bt
|
||||
\[
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -3187,11 +3187,11 @@ our network must solve, is
|
||||
\]
|
||||
!et
|
||||
|
||||
or in terms of $P_{\text{hidden} }$ and $P_{\text{output} }$
|
||||
or in terms of $P_{\mathrm{hidden} }$ and $P_{\mathrm{output} }$
|
||||
|
||||
!bt
|
||||
\[
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -3211,30 +3211,30 @@ network. Having an implementation of a neural network at hand, an
|
||||
extension of it into a deep neural network would (hopefully) be
|
||||
painless.
|
||||
|
||||
For simplicity, we assume that the input is an array $\vec x =
|
||||
(x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural
|
||||
For simplicity, we assume that the input is an array
|
||||
$\hat{x}= (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural
|
||||
network should find $P$ such that it fulfills (ref{eq:min}).
|
||||
|
||||
!split
|
||||
===== Feedforward =====
|
||||
|
||||
First, a feedforward of the inputs must be done. This means that $\vec
|
||||
x$ must be passed through an input layer, a hidden layer and a output
|
||||
First, a feedforward of the inputs must be done. This means that $\hat{x}$
|
||||
must be passed through an input layer, a hidden layer and a output
|
||||
layer. The input layer in this case, does not need to process the
|
||||
data any further. The input layer will consist of $N_{\text{input} }$
|
||||
data any further. The input layer will consist of $N_{\mathrm{input} }$
|
||||
neurons, passing its element to each neuron in the hidden layer. The
|
||||
number of neurons in the hidden layer will be $N_{\text{hidden} }$.
|
||||
number of neurons in the hidden layer will be $N_{\mathrm{hidden} }$.
|
||||
|
||||
For the $i$-th in the hidden layer with weight $w_i^{\text{hidden} }$
|
||||
and bias $b_i^{\text{hidden} }$, the weighting from the $j$-th neuron
|
||||
For the $i$-th in the hidden layer with weight $w_i^{\mathrm{hidden} }$
|
||||
and bias $b_i^{\mathrm{hidden} }$, the weighting from the $j$-th neuron
|
||||
at the input layer is:
|
||||
|
||||
!bt
|
||||
\begin{aligned}
|
||||
z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
|
||||
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
@@ -3250,26 +3250,26 @@ x_j
|
||||
The result after weighting the input at the $i$-th hidden neuron can be written as a vector:
|
||||
!bt
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
|
||||
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
x_1 & x_2 & \dots & x_N
|
||||
\end{pmatrix} \\
|
||||
&= \vec{p}_{i, \text{hidden}}^T X
|
||||
&= \hat{p}_{i, \mathrm{hidden}}^T X
|
||||
\end{aligned}
|
||||
!et
|
||||
|
||||
It is the vector $\vec{p}_{i, \text{hidden}}^T$ that defines each row
|
||||
in $P_{\text{hidden} }$, which contains the weights for the neural
|
||||
It is the vector $\hat{p}_{i, \mathrm{hidden}}^T$ that defines each row
|
||||
in $P_{\mathrm{hidden} }$, which contains the weights for the neural
|
||||
network to minimize according to (ref{eq:min}).
|
||||
|
||||
After having found $\vec{z}_{i}^{\text{hidden}} $ for every neuron $i$
|
||||
After having found $\hat{z}_{i}^{\mathrm{hidden}} $ for every neuron $i$
|
||||
in the hidden layer, the vector will be sent to an activation function
|
||||
$a_i(\vec{z})$. In this example, the sigmoid function has been used:
|
||||
$a_i(\hat{z})$. In this example, the sigmoid function has been used:
|
||||
|
||||
$$
|
||||
f(z) = \frac{1}{1 + \exp{(-z)}}.
|
||||
@@ -3279,18 +3279,18 @@ $$
|
||||
!split
|
||||
===== Output =====
|
||||
|
||||
The output $\vec{x}_i^{\text{hidden} }$from each $i$-th hidden neuron is:
|
||||
The output $\hat{x}_i^{\mathrm{hidden}}$from each $i$-th hidden neuron is:
|
||||
|
||||
$$
|
||||
\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
|
||||
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
|
||||
$$
|
||||
|
||||
The outputs $\vec{x}_i^{\text{hidden} } $ are then sent to the output layer.
|
||||
The outputs $\hat{x}_i^{\mathrm{hidden} } $ are then sent to the output layer.
|
||||
|
||||
The output layer consist of one neuron in this case, and combines the
|
||||
output from each of the neurons in the hidden layers. The output layer
|
||||
combines the results from the hidden layer using some weights $
|
||||
w_i^{\text{output}}$ and biases $b_i^{\text{output}}$. In this case,
|
||||
w_i^{\mathrm{output}}$ and biases $b_i^{\mathrm{output}}$. In this case,
|
||||
it is assumes that the number of neurons in the output layer is one.
|
||||
|
||||
The procedure of weigthing the output neuron $j$ in the hidden layer
|
||||
@@ -3299,34 +3299,34 @@ layer described previously.
|
||||
|
||||
!bt
|
||||
\begin{aligned}
|
||||
z_{1,j}^{\text{output}} & =
|
||||
z_{1,j}^{\mathrm{output}} & =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{\text{hidden}}
|
||||
\hat{x}_j^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
!et
|
||||
|
||||
Expressing $z_{1,j}^{\text{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
Expressing $z_{1,j}^{\mathrm{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
|
||||
$$
|
||||
\vec{z}_{1}^{\text{output}} =
|
||||
\hat{z}_{1}^{\mathrm{output}} =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
|
||||
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
In this case we seek a continous range of values since we are
|
||||
approximating a function. This means that after computing
|
||||
$\vec{z}_{1}^{\text{output}}$ the neural network has finished its
|
||||
feedforward step, and $\vec{z}_{1}^{\text{output}}$ is the final
|
||||
$\hat{z}_{1}^{\mathrm{output}}$ the neural network has finished its
|
||||
feedforward step, and $\hat{z}_{1}^{\mathrm{output}}$ is the final
|
||||
output of the network.
|
||||
|
||||
|
||||
@@ -3442,12 +3442,12 @@ goes as follows:
|
||||
|
||||
!bt
|
||||
\[
|
||||
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
\]
|
||||
!et
|
||||
|
||||
for a number of iterations or until $ \big|\big| \vec
|
||||
\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
given tolerance.
|
||||
|
||||
The value of $\lambda$ decides how large steps the algorithm must take
|
||||
@@ -3461,16 +3461,16 @@ the elements in $\vec \omega$.
|
||||
|
||||
In our case, we have to minimize the cost function $c(x, P)$ with
|
||||
respect to the two sets of weights and bisases, that is for the hidden
|
||||
layer $P_{\text{hidden} }$ and for the ouput layer $P_{\text{output}
|
||||
layer $P_{\mathrm{hidden} }$ and for the ouput layer $P_{\mathrm{output}
|
||||
}$ .
|
||||
|
||||
This means that $P_{\text{hidden} }$ and $P_{\text{output} }$ is
|
||||
This means that $P_{\mathrm{hidden} }$ and $P_{\mathrm{output} }$ is
|
||||
updated by
|
||||
|
||||
!bt
|
||||
\begin{aligned}
|
||||
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
|
||||
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
|
||||
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
|
||||
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
|
||||
\end{aligned}
|
||||
!et
|
||||
|
||||
@@ -3519,13 +3519,13 @@ def solve_ode_neural_network(x, num_neurons_hidden, num_iter, lmb):
|
||||
|
||||
As previously stated, a Deep Neural Network (DNN) follows the same
|
||||
concept of a neural network, but having more than one hidden
|
||||
layer. Suppose that the network has $N_{\text{hidden}}$ hidden layers
|
||||
where the $l$-th layer has $N_{\text{hidden}}^{(l)}$ neurons. The
|
||||
layer. Suppose that the network has $N_{\mathrm{hidden}}$ hidden layers
|
||||
where the $l$-th layer has $N_{\mathrm{hidden}}^{(l)}$ neurons. The
|
||||
input is still assumed to be an array of size $1 \times N$. The
|
||||
network must now try to optimalize its output with respect to the
|
||||
collection of weigths and biases $P = \big\{P_{\text{input} }, \
|
||||
P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\}$.
|
||||
collection of weigths and biases $P = \big\{P_{\mathrm{input} }, \
|
||||
P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\}$.
|
||||
|
||||
|
||||
!split
|
||||
@@ -3534,39 +3534,39 @@ P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\}$.
|
||||
The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer.
|
||||
|
||||
The $i$-th neuron at layer $l$ recieves the result
|
||||
$\vec{x}_j^{(l-1),\text{hidden} }$ from the $j$-th neuron at layer
|
||||
$\hat{x}_j^{(l-1),\mathrm{hidden} }$ from the $j$-th neuron at layer
|
||||
$l-1$. The $i$-th neuron at layer $l$ weights all of the elements in
|
||||
$\vec{x}_j^{(l-1),\text{hidden} }$ with a weight vector $\vec
|
||||
w_{i,j}^{(l), \ \text{hidden} }$ with as many weigths as there are
|
||||
elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias
|
||||
$b_i^{(l), \ \text{hidden} }$:
|
||||
$\hat{x}_j^{(l-1),\mathrm{hidden} }$ with a weight vector $\vec
|
||||
w_{i,j}^{(l), \ \mathrm{hidden} }$ with as many weigths as there are
|
||||
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
|
||||
$b_i^{(l), \ \mathrm{hidden} }$:
|
||||
|
||||
!bt
|
||||
\begin{aligned}
|
||||
z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
|
||||
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{(l-1),\text{hidden} }
|
||||
\hat{x}_j^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
!et
|
||||
|
||||
The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\vec{z}_{i}^{(l),\ \text{hidden}}$:
|
||||
The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\hat{z}_{i}^{(l),\ \mathrm{hidden}}$:
|
||||
|
||||
!bt
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
|
||||
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
|
||||
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
!et
|
||||
|
||||
Reference in New Issue
Block a user