lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

deleting unnecessary material

Browse Source
This commit is contained in:
mhjensen
2018-10-19 06:43:54 +02:00
parent 97ee7402a8
commit d5af6f12c1
113 changed files with 2879 additions and 4700 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/NeuralNet/NeuralNet.do.txt
+6 -157
View File
@@ -3214,120 +3214,9 @@ painless.
For simplicity, we assume that the input is an array
$\hat{x}= (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural
network should find $P$ such that it fulfills (ref{eq:min}).
All the ingredients discussed earlier, from the activation function, hidden layers and their weights, biases etc
are included below.
!split
===== Feedforward =====
First, a feedforward of the inputs must be done. This means that $\hat{x}$
must be passed through an input layer, a hidden layer and a output
layer. The input layer in this case, does not need to process the
data any further. The input layer will consist of $N_{\mathrm{input} }$
neurons, passing its element to each neuron in the hidden layer. The
number of neurons in the hidden layer will be $N_{\mathrm{hidden} }$.
For the $i$-th in the hidden layer with weight $w_i^{\mathrm{hidden} }$
and bias $b_i^{\mathrm{hidden} }$, the weighting from the $j$-th neuron
at the input layer is:
!bt
\begin{aligned}
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
&=
\begin{pmatrix}
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 \\
x_j
\end{pmatrix}
\end{aligned}
!et
!split
===== Result after weighting =====
The result after weighting the input at the $i$-th hidden neuron can be written as a vector:
!bt
\begin{aligned}
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
&=
\begin{pmatrix}
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
x_1 & x_2 & \dots & x_N
\end{pmatrix} \\
&= \hat{p}_{i, \mathrm{hidden}}^T X
\end{aligned}
!et
It is the vector $\hat{p}_{i, \mathrm{hidden}}^T$ that defines each row
in $P_{\mathrm{hidden} }$, which contains the weights for the neural
network to minimize according to (ref{eq:min}).
After having found $\hat{z}_{i}^{\mathrm{hidden}} $ for every neuron $i$
in the hidden layer, the vector will be sent to an activation function
$a_i(\hat{z})$. In this example, the sigmoid function has been used:
$$
f(z) = \frac{1}{1 + \exp{(-z)}}.
$$
!split
===== Output =====
The output $\hat{x}_i^{\mathrm{hidden}}$from each $i$-th hidden neuron is:
$$
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
$$
The outputs $\hat{x}_i^{\mathrm{hidden} } $ are then sent to the output layer.
The output layer consist of one neuron in this case, and combines the
output from each of the neurons in the hidden layers. The output layer
combines the results from the hidden layer using some weights $
w_i^{\mathrm{output}}$ and biases $b_i^{\mathrm{output}}$. In this case,
it is assumes that the number of neurons in the output layer is one.
The procedure of weigthing the output neuron $j$ in the hidden layer
to the $i$-th neuron in the output layer is similar as for the hidden
layer described previously.
!bt
\begin{aligned}
z_{1,j}^{\mathrm{output}} & =
\begin{pmatrix}
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 \\
\hat{x}_j^{\mathrm{hidden}}
\end{pmatrix}
\end{aligned}
!et
Expressing $z_{1,j}^{\mathrm{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:
$$
\hat{z}_{1}^{\mathrm{output}} =
\begin{pmatrix}
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
\end{pmatrix}
$$
In this case we seek a continous range of values since we are
approximating a function. This means that after computing
$\hat{z}_{1}^{\mathrm{output}}$ the neural network has finished its
feedforward step, and $\hat{z}_{1}^{\mathrm{output}}$ is the final
output of the network.
!split
@@ -3381,7 +3270,7 @@ def neural_network(params, x):
!split
===== Backpropagation =====
Now that feedforward can be done, the next step is to decide how the
Now that the feedforward can be done, the next step is to decide how the
parameters should change such that they minimize the cost function.
Recall that the chosen cost function for this problem is
@@ -3468,10 +3357,10 @@ This means that $P_{\mathrm{hidden} }$ and $P_{\mathrm{output} }$ is
updated by
!bt
\begin{aligned}
\begin{align}
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
\end{aligned}
\end{align}
!et
This might look like a cumberstone to set up the correct expression
@@ -3528,47 +3417,7 @@ P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \
P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\}$.
!split
===== Feed forward again =====
The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer.
The $i$-th neuron at layer $l$ recieves the result
$\hat{x}_j^{(l-1),\mathrm{hidden} }$ from the $j$-th neuron at layer
$l-1$. The $i$-th neuron at layer $l$ weights all of the elements in
$\hat{x}_j^{(l-1),\mathrm{hidden} }$ with a weight vector $w_{i,j}^{(l), \ \mathrm{hidden}}$ with as many weigths as there are
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
$b_i^{(l), \ \mathrm{hidden} }$:
!bt
\begin{aligned}
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
&=
\begin{pmatrix}
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 \\
\hat{x}_j^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
!et
The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\hat{z}_{i}^{(l),\ \mathrm{hidden}}$:
!bt
\begin{aligned}
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
&=
\begin{pmatrix}
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
!et
!split
===== The final parts of the code =====
@@ -3710,7 +3559,7 @@ The code below solves the ODE using a neural network. The number of
values for the input $\vec x$ is 10, number of hidden neurons in the
hidden layer being 10 and th step size used in gradien descent
$\lambda = 0.001$. The program updates the weights and biases in the
network *num_iter* times. Finally, it plots the results from using the
network for a given number of iterations. Finally, it plots the results from using the
neural network along with the analytical solution.