more boring typos
This commit is contained in:
mhjensen
@@ -3077,7 +3077,7 @@ depth respectively).
|
||||
The neurons in a layer will
|
||||
only be connected to a small region of the layer before it, instead of
|
||||
all of the neurons in a fully-connected manner. Moreover, the final
|
||||
output layer could for this specific image have dimensions \( 1\times 1 times 10 \),
|
||||
output layer could for this specific image have dimensions \( 1\times 1 \times 10 \),
|
||||
because by the
|
||||
end of the CNN architecture we will reduce the full image into a
|
||||
single vector of class scores, arranged along the depth
|
||||
@@ -3708,17 +3708,17 @@ and biases for each layer.
|
||||
|
||||
<p>
|
||||
It is assumed that there are no weights and
|
||||
bias at the input layer, so \( P = \{ P_{\text{hidden}},
|
||||
P_{\text{output}} \} \). If there are \( N_{\text{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\text{hidden}} \) is an \( N_{\text{hidden} }
|
||||
bias at the input layer, so \( P = \{ P_{\mathrm{hidden}},
|
||||
P_{\mathrm{output}} \} \). If there are \( N_{\mathrm{hidden} } \) neurons in
|
||||
the hidden layer, then \( P_{\mathrm{hidden}} \) is an \( N_{\mathrm{hidden} }
|
||||
\times 2 \) matrix.
|
||||
|
||||
<p>
|
||||
The first column in \( P_{\text{hidden} } \) represents
|
||||
The first column in \( P_{\mathrm{hidden} } \) represents
|
||||
the bias for each neuron in the hidden layer and the second column
|
||||
represents the weigths for each neuron. If there are \( N_{\text{output}
|
||||
} \) neurons in the output layer, then \( P_{\text{output}} \) is a
|
||||
\( N_{\text{output} } \times (1 + N_{\text{hidden} }) \) matrix. Its first
|
||||
represents the weigths for each neuron. If there are \( N_{\mathrm{output}
|
||||
} \) neurons in the output layer, then \( P_{\mathrm{output}} \) is a
|
||||
\( N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} }) \) matrix. Its first
|
||||
column represents the bias of each neuron and the remaining columns
|
||||
represents the weights to each neuron.
|
||||
|
||||
@@ -3794,7 +3794,7 @@ $$
|
||||
or, in terms of weights and biases for each layer:
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -3818,10 +3818,10 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
or in terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \)
|
||||
or in terms of \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \)
|
||||
|
||||
$$
|
||||
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
|
||||
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -3844,8 +3844,8 @@ extension of it into a deep neural network would (hopefully) be
|
||||
painless.
|
||||
|
||||
<p>
|
||||
For simplicity, we assume that the input is an array \( \vec x =
|
||||
(x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
For simplicity, we assume that the input is an array
|
||||
\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
|
||||
network should find \( P \) such that it fulfills \eqref{eq:min}.
|
||||
|
||||
<p>
|
||||
@@ -3854,24 +3854,24 @@ network should find \( P \) such that it fulfills \eqref{eq:min}.
|
||||
<h2 id="___sec93">Feedforward </h2>
|
||||
|
||||
<p>
|
||||
First, a feedforward of the inputs must be done. This means that \( \vec
|
||||
x \) must be passed through an input layer, a hidden layer and a output
|
||||
First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
|
||||
must be passed through an input layer, a hidden layer and a output
|
||||
layer. The input layer in this case, does not need to process the
|
||||
data any further. The input layer will consist of \( N_{\text{input} } \)
|
||||
data any further. The input layer will consist of \( N_{\mathrm{input} } \)
|
||||
neurons, passing its element to each neuron in the hidden layer. The
|
||||
number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
|
||||
number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
|
||||
|
||||
<p>
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \)
|
||||
and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
|
||||
and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
|
||||
at the input layer is:
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
|
||||
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
@@ -3889,28 +3889,28 @@ $$
|
||||
The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector:
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
|
||||
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{\text{hidden}} & w_i^{\text{hidden}}
|
||||
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
x_1 & x_2 & \dots & x_N
|
||||
\end{pmatrix} \\
|
||||
&= \vec{p}_{i, \text{hidden}}^T X
|
||||
&= \hat{p}_{i, \mathrm{hidden}}^T X
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
It is the vector \( \vec{p}_{i, \text{hidden}}^T \) that defines each row
|
||||
in \( P_{\text{hidden} } \), which contains the weights for the neural
|
||||
It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
|
||||
in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
|
||||
network to minimize according to \eqref{eq:min}.
|
||||
|
||||
<p>
|
||||
After having found \( \vec{z}_{i}^{\text{hidden}} \) for every neuron \( i \)
|
||||
After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
|
||||
in the hidden layer, the vector will be sent to an activation function
|
||||
\( a_i(\vec{z}) \). In this example, the sigmoid function has been used:
|
||||
\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
|
||||
|
||||
$$
|
||||
f(z) = \frac{1}{1 + \exp{(-z)}}.
|
||||
@@ -3922,20 +3922,20 @@ $$
|
||||
<h2 id="___sec95">Output </h2>
|
||||
|
||||
<p>
|
||||
The output $\vec{x}_i^{\text{hidden} }$from each \( i \)-th hidden neuron is:
|
||||
The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
|
||||
|
||||
$$
|
||||
\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
|
||||
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
|
||||
$$
|
||||
|
||||
<p>
|
||||
The outputs \( \vec{x}_i^{\text{hidden} } \) are then sent to the output layer.
|
||||
The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
|
||||
|
||||
<p>
|
||||
The output layer consist of one neuron in this case, and combines the
|
||||
output from each of the neurons in the hidden layers. The output layer
|
||||
combines the results from the hidden layer using some weights \(
|
||||
w_i^{\text{output}} \) and biases \( b_i^{\text{output}} \). In this case,
|
||||
w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
|
||||
it is assumes that the number of neurons in the output layer is one.
|
||||
|
||||
<p>
|
||||
@@ -3945,36 +3945,36 @@ layer described previously.
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{1,j}^{\text{output}} & =
|
||||
z_{1,j}^{\mathrm{output}} & =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{\text{hidden}}
|
||||
\hat{x}_j^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
|
||||
|
||||
$$
|
||||
\vec{z}_{1}^{\text{output}} =
|
||||
\hat{z}_{1}^{\mathrm{output}} =
|
||||
\begin{pmatrix}
|
||||
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
|
||||
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
|
||||
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
<p>
|
||||
In this case we seek a continous range of values since we are
|
||||
approximating a function. This means that after computing
|
||||
\( \vec{z}_{1}^{\text{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \vec{z}_{1}^{\text{output}} \) is the final
|
||||
\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
|
||||
feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
|
||||
output of the network.
|
||||
|
||||
<p>
|
||||
@@ -4100,12 +4100,12 @@ function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \),
|
||||
goes as follows:
|
||||
|
||||
$$
|
||||
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
|
||||
$$
|
||||
|
||||
<p>
|
||||
for a number of iterations or until $ \big|\big| \vec
|
||||
\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
|
||||
given tolerance.
|
||||
|
||||
<p>
|
||||
@@ -4122,17 +4122,17 @@ the elements in \( \vec \omega \).
|
||||
<p>
|
||||
In our case, we have to minimize the cost function \( c(x, P) \) with
|
||||
respect to the two sets of weights and bisases, that is for the hidden
|
||||
layer \( P_{\text{hidden} } \) and for the ouput layer \( P_{\text{output}
|
||||
layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output}
|
||||
} \) .
|
||||
|
||||
<p>
|
||||
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is
|
||||
This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
|
||||
updated by
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
|
||||
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
|
||||
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
|
||||
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
@@ -4185,13 +4185,13 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
|
||||
<p>
|
||||
As previously stated, a Deep Neural Network (DNN) follows the same
|
||||
concept of a neural network, but having more than one hidden
|
||||
layer. Suppose that the network has \( N_{\text{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\text{hidden}}^{(l)} \) neurons. The
|
||||
layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers
|
||||
where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The
|
||||
input is still assumed to be an array of size \( 1 \times N \). The
|
||||
network must now try to optimalize its output with respect to the
|
||||
collection of weigths and biases \( P = \big\{P_{\text{input} }, \
|
||||
P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\} \).
|
||||
collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \
|
||||
P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
|
||||
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -4203,40 +4203,40 @@ The feedforward step is similar to as for the neural netowork, but now consideri
|
||||
|
||||
<p>
|
||||
The \( i \)-th neuron at layer \( l \) recieves the result
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
|
||||
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
|
||||
\( \vec{x}_j^{(l-1),\text{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \text{hidden} } \) with as many weigths as there are
|
||||
elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \text{hidden} } \):
|
||||
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec
|
||||
w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are
|
||||
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
|
||||
\( b_i^{(l), \ \mathrm{hidden} } \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
|
||||
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 \\
|
||||
\vec{x}_j^{(l-1),\text{hidden} }
|
||||
\hat{x}_j^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
<p>
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \vec{z}_{i}^{(l),\ \text{hidden}} \):
|
||||
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
|
||||
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
|
||||
&=
|
||||
\begin{pmatrix}
|
||||
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
|
||||
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
|
||||
\end{pmatrix}
|
||||
\begin{pmatrix}
|
||||
1 & 1 & \dots & 1 \\
|
||||
\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
|
||||
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
|
||||
\end{pmatrix}
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
Reference in New Issue
Block a user