lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

more boring typos

Browse Source
This commit is contained in:
mhjensen
2018-10-19 06:07:18 +02:00
parent a067cf3e04
commit 092fdecd9b
18 changed files with 384 additions and 384 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/NeuralNet/NeuralNet.do.txt
+64 -64
View File
@@ -2517,7 +2517,7 @@ depth respectively).
The neurons in a layer will
only be connected to a small region of the layer before it, instead of
all of the neurons in a fully-connected manner. Moreover, the final
output layer could for this specific image have dimensions $1\times 1 times 10$,
output layer could for this specific image have dimensions $1\times 1 \times 10$,
because by the
end of the CNN architecture we will reduce the full image into a
single vector of class scores, arranged along the depth
@@ -3088,16 +3088,16 @@ neural network $N(x,P)$ with $P $ being the collection of the weights
and biases for each layer.
It is assumed that there are no weights and
bias at the input layer, so $P = \{ P_{\text{hidden}},
P_{\text{output}} \}$. If there are $N_{\text{hidden} }$ neurons in
the hidden layer, then $P_{\text{hidden}}$ is an $N_{\text{hidden} }
bias at the input layer, so $P = \{ P_{\mathrm{hidden}},
P_{\mathrm{output}} \}$. If there are $N_{\mathrm{hidden} }$ neurons in
the hidden layer, then $P_{\mathrm{hidden}}$ is an $N_{\mathrm{hidden} }
\times 2$ matrix.
The first column in $P_{\text{hidden} }$ represents
The first column in $P_{\mathrm{hidden} }$ represents
the bias for each neuron in the hidden layer and the second column
represents the weigths for each neuron. If there are $N_{\text{output}
}$ neurons in the output layer, then $P_{\text{output}} $ is a
$N_{\text{output} } \times (1 + N_{\text{hidden} })$ matrix. Its first
represents the weigths for each neuron. If there are $N_{\mathrm{output}
}$ neurons in the output layer, then $P_{\mathrm{output}} $ is a
$N_{\mathrm{output} } \times (1 + N_{\mathrm{hidden} })$ matrix. Its first
column represents the bias of each neuron and the remaining columns
represents the weights to each neuron.
@@ -3163,7 +3163,7 @@ or, in terms of weights and biases for each layer:
!bt
\[
\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\}
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }}\Big\{ \big(g_t'(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) - ( -\gamma g_t(x, \{ P_{\mathrm{hidden} }, P_{\mathrm{output} }\}) \big)^2 \Big\}
\]
!et
@@ -3187,11 +3187,11 @@ our network must solve, is
\]
!et
or in terms of $P_{\text{hidden} }$ and $P_{\text{output} }$
or in terms of $P_{\mathrm{hidden} }$ and $P_{\mathrm{output} }$
!bt
\[
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\})
\min_{P_{\mathrm{hidden} }, \ P_{\mathrm{output} }} c(x, \{P_{\mathrm{hidden} }, P_{\mathrm{output} }\})
\]
!et
@@ -3211,30 +3211,30 @@ network. Having an implementation of a neural network at hand, an
extension of it into a deep neural network would (hopefully) be
painless.
For simplicity, we assume that the input is an array $\vec x =
(x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural
For simplicity, we assume that the input is an array
$\hat{x}= (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural
network should find $P$ such that it fulfills (ref{eq:min}).
!split
===== Feedforward =====
First, a feedforward of the inputs must be done. This means that $\vec
x$ must be passed through an input layer, a hidden layer and a output
First, a feedforward of the inputs must be done. This means that $\hat{x}$
must be passed through an input layer, a hidden layer and a output
layer. The input layer in this case, does not need to process the
data any further. The input layer will consist of $N_{\text{input} }$
data any further. The input layer will consist of $N_{\mathrm{input} }$
neurons, passing its element to each neuron in the hidden layer. The
number of neurons in the hidden layer will be $N_{\text{hidden} }$.
number of neurons in the hidden layer will be $N_{\mathrm{hidden} }$.
For the $i$-th in the hidden layer with weight $w_i^{\text{hidden} }$
and bias $b_i^{\text{hidden} }$, the weighting from the $j$-th neuron
For the $i$-th in the hidden layer with weight $w_i^{\mathrm{hidden} }$
and bias $b_i^{\mathrm{hidden} }$, the weighting from the $j$-th neuron
at the input layer is:
!bt
\begin{aligned}
z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
&=
\begin{pmatrix}
b_i^{\text{hidden}} & w_i^{\text{hidden}}
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 \\
@@ -3250,26 +3250,26 @@ x_j
The result after weighting the input at the $i$-th hidden neuron can be written as a vector:
!bt
\begin{aligned}
\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
&=
\begin{pmatrix}
b_i^{\text{hidden}} & w_i^{\text{hidden}}
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
x_1 & x_2 & \dots & x_N
\end{pmatrix} \\
&= \vec{p}_{i, \text{hidden}}^T X
&= \hat{p}_{i, \mathrm{hidden}}^T X
\end{aligned}
!et
It is the vector $\vec{p}_{i, \text{hidden}}^T$ that defines each row
in $P_{\text{hidden} }$, which contains the weights for the neural
It is the vector $\hat{p}_{i, \mathrm{hidden}}^T$ that defines each row
in $P_{\mathrm{hidden} }$, which contains the weights for the neural
network to minimize according to (ref{eq:min}).
After having found $\vec{z}_{i}^{\text{hidden}} $ for every neuron $i$
After having found $\hat{z}_{i}^{\mathrm{hidden}} $ for every neuron $i$
in the hidden layer, the vector will be sent to an activation function
$a_i(\vec{z})$. In this example, the sigmoid function has been used:
$a_i(\hat{z})$. In this example, the sigmoid function has been used:
$$
f(z) = \frac{1}{1 + \exp{(-z)}}.
@@ -3279,18 +3279,18 @@ $$
!split
===== Output =====
The output $\vec{x}_i^{\text{hidden} }$from each $i$-th hidden neuron is:
The output $\hat{x}_i^{\mathrm{hidden}}$from each $i$-th hidden neuron is:
$$
\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big).
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
$$
The outputs $\vec{x}_i^{\text{hidden} } $ are then sent to the output layer.
The outputs $\hat{x}_i^{\mathrm{hidden} } $ are then sent to the output layer.
The output layer consist of one neuron in this case, and combines the
output from each of the neurons in the hidden layers. The output layer
combines the results from the hidden layer using some weights $
w_i^{\text{output}}$ and biases $b_i^{\text{output}}$. In this case,
w_i^{\mathrm{output}}$ and biases $b_i^{\mathrm{output}}$. In this case,
it is assumes that the number of neurons in the output layer is one.
The procedure of weigthing the output neuron $j$ in the hidden layer
@@ -3299,34 +3299,34 @@ layer described previously.
!bt
\begin{aligned}
z_{1,j}^{\text{output}} & =
z_{1,j}^{\mathrm{output}} & =
\begin{pmatrix}
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 \\
\vec{x}_j^{\text{hidden}}
\hat{x}_j^{\mathrm{hidden}}
\end{pmatrix}
\end{aligned}
!et
Expressing $z_{1,j}^{\text{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:
Expressing $z_{1,j}^{\mathrm{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:
$$
\vec{z}_{1}^{\text{output}} =
\hat{z}_{1}^{\mathrm{output}} =
\begin{pmatrix}
b_1^{\text{output}} & \vec{w}_1^{\text{output}}
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}}
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
\end{pmatrix}
$$
In this case we seek a continous range of values since we are
approximating a function. This means that after computing
$\vec{z}_{1}^{\text{output}}$ the neural network has finished its
feedforward step, and $\vec{z}_{1}^{\text{output}}$ is the final
$\hat{z}_{1}^{\mathrm{output}}$ the neural network has finished its
feedforward step, and $\hat{z}_{1}^{\mathrm{output}}$ is the final
output of the network.
@@ -3442,12 +3442,12 @@ goes as follows:
!bt
\[
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
\]
!et
for a number of iterations or until $ \big|\big| \vec
\omega_{\text{new} } - \vec \omega \big|\big|$ is smaller than some
\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
given tolerance.
The value of $\lambda$ decides how large steps the algorithm must take
@@ -3461,16 +3461,16 @@ the elements in $\vec \omega$.
In our case, we have to minimize the cost function $c(x, P)$ with
respect to the two sets of weights and bisases, that is for the hidden
layer $P_{\text{hidden} }$ and for the ouput layer $P_{\text{output}
layer $P_{\mathrm{hidden} }$ and for the ouput layer $P_{\mathrm{output}
}$ .
This means that $P_{\text{hidden} }$ and $P_{\text{output} }$ is
This means that $P_{\mathrm{hidden} }$ and $P_{\mathrm{output} }$ is
updated by
!bt
\begin{aligned}
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P)
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
\end{aligned}
!et
@@ -3519,13 +3519,13 @@ def solve_ode_neural_network(x, num_neurons_hidden, num_iter, lmb):
As previously stated, a Deep Neural Network (DNN) follows the same
concept of a neural network, but having more than one hidden
layer. Suppose that the network has $N_{\text{hidden}}$ hidden layers
where the $l$-th layer has $N_{\text{hidden}}^{(l)}$ neurons. The
layer. Suppose that the network has $N_{\mathrm{hidden}}$ hidden layers
where the $l$-th layer has $N_{\mathrm{hidden}}^{(l)}$ neurons. The
input is still assumed to be an array of size $1 \times N$. The
network must now try to optimalize its output with respect to the
collection of weigths and biases $P = \big\{P_{\text{input} }, \
P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \
P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\}$.
collection of weigths and biases $P = \big\{P_{\mathrm{input} }, \
P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\}$.
!split
@@ -3534,39 +3534,39 @@ P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\}$.
The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer.
The $i$-th neuron at layer $l$ recieves the result
$\vec{x}_j^{(l-1),\text{hidden} }$ from the $j$-th neuron at layer
$\hat{x}_j^{(l-1),\mathrm{hidden} }$ from the $j$-th neuron at layer
$l-1$. The $i$-th neuron at layer $l$ weights all of the elements in
$\vec{x}_j^{(l-1),\text{hidden} }$ with a weight vector $\vec
w_{i,j}^{(l), \ \text{hidden} }$ with as many weigths as there are
elements in$\vec{x}_j^{(l-1),\text{hidden} }$, and adds a bias
$b_i^{(l), \ \text{hidden} }$:
$\hat{x}_j^{(l-1),\mathrm{hidden} }$ with a weight vector $\vec
w_{i,j}^{(l), \ \mathrm{hidden} }$ with as many weigths as there are
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
$b_i^{(l), \ \mathrm{hidden} }$:
!bt
\begin{aligned}
z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
&=
\begin{pmatrix}
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 \\
\vec{x}_j^{(l-1),\text{hidden} }
\hat{x}_j^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
!et
The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\vec{z}_{i}^{(l),\ \text{hidden}}$:
The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\hat{z}_{i}^{(l),\ \mathrm{hidden}}$:
!bt
\begin{aligned}
\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
&=
\begin{pmatrix}
b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} }
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
!et