lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

deleting unnecessary material

Browse Source
This commit is contained in:
mhjensen
2018-10-19 06:43:54 +02:00
parent 97ee7402a8
commit d5af6f12c1
113 changed files with 2879 additions and 4700 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/NeuralNet.html
+31 -207
View File
@@ -210,24 +210,20 @@ div { text-align: justify; text-justify: inter-word; }
('Reformulating the problem', 2, None, '___sec90'),
('Estimating errors', 2, None, '___sec91'),
('Creating a simple Deep Neural Net', 2, None, '___sec92'),
('Feedforward', 2, None, '___sec93'),
('Result after weighting', 2, None, '___sec94'),
('Output', 2, None, '___sec95'),
('Setting up the code, feed forward part', 2, None, '___sec96'),
('Backpropagation', 2, None, '___sec97'),
('Gradient Descent', 2, None, '___sec98'),
('More on GD and cost function', 2, None, '___sec99'),
('Setting up the code, feed forward part', 2, None, '___sec93'),
('Backpropagation', 2, None, '___sec94'),
('Gradient Descent', 2, None, '___sec95'),
('More on GD and cost function', 2, None, '___sec96'),
('An implementation of a Deep Neural Network',
2,
None,
'___sec100'),
('Feed forward again', 2, None, '___sec101'),
('The final parts of the code', 2, None, '___sec102'),
('And adding Back propagation', 2, None, '___sec103'),
('Solving the ODE', 2, None, '___sec104'),
('Using neural network', 2, None, '___sec105'),
('Using a deep neural network', 2, None, '___sec106'),
('Wrapping it up', 2, None, '___sec107')]}
'___sec97'),
('The final parts of the code', 2, None, '___sec98'),
('And adding Back propagation', 2, None, '___sec99'),
('Solving the ODE', 2, None, '___sec100'),
('Using neural network', 2, None, '___sec101'),
('Using a deep neural network', 2, None, '___sec102'),
('Wrapping it up', 2, None, '___sec103')]}
end of tocinfo -->
<body>
@@ -3847,140 +3843,13 @@ painless.
For simplicity, we assume that the input is an array
\( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural
network should find \( P \) such that it fulfills \eqref{eq:min}.
All the ingredients discussed earlier, from the activation function, hidden layers and their weights, biases etc
are included below.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec93">Feedforward </h2>
<p>
First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
must be passed through an input layer, a hidden layer and a output
layer. The input layer in this case, does not need to process the
data any further. The input layer will consist of \( N_{\mathrm{input} } \)
neurons, passing its element to each neuron in the hidden layer. The
number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
<p>
For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
at the input layer is:
$$
\begin{aligned}
z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
&=
\begin{pmatrix}
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 \\
x_j
\end{pmatrix}
\end{aligned}
$$
<p>
<!-- !split -->
<h2 id="___sec94">Result after weighting </h2>
<p>
The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector:
$$
\begin{aligned}
\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
&=
\begin{pmatrix}
b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
x_1 & x_2 & \dots & x_N
\end{pmatrix} \\
&= \hat{p}_{i, \mathrm{hidden}}^T X
\end{aligned}
$$
<p>
It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
network to minimize according to \eqref{eq:min}.
<p>
After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
in the hidden layer, the vector will be sent to an activation function
\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
$$
f(z) = \frac{1}{1 + \exp{(-z)}}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec95">Output </h2>
<p>
The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
$$
\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
$$
<p>
The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
<p>
The output layer consist of one neuron in this case, and combines the
output from each of the neurons in the hidden layers. The output layer
combines the results from the hidden layer using some weights \(
w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
it is assumes that the number of neurons in the output layer is one.
<p>
The procedure of weigthing the output neuron \( j \) in the hidden layer
to the \( i \)-th neuron in the output layer is similar as for the hidden
layer described previously.
$$
\begin{aligned}
z_{1,j}^{\mathrm{output}} & =
\begin{pmatrix}
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 \\
\hat{x}_j^{\mathrm{hidden}}
\end{pmatrix}
\end{aligned}
$$
<p>
Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
$$
\hat{z}_{1}^{\mathrm{output}} =
\begin{pmatrix}
b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
\end{pmatrix}
\begin{pmatrix}
1 &amp; 1 & \dots &amp; 1 \\
\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
\end{pmatrix}
$$
<p>
In this case we seek a continous range of values since we are
approximating a function. This means that after computing
\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
output of the network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec96">Setting up the code, feed forward part </h2>
<h2 id="___sec93">Setting up the code, feed forward part </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -4031,10 +3900,10 @@ output of the network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec97">Backpropagation </h2>
<h2 id="___sec94">Backpropagation </h2>
<p>
Now that feedforward can be done, the next step is to decide how the
Now that the feedforward can be done, the next step is to decide how the
parameters should change such that they minimize the cost function.
<p>
@@ -4088,7 +3957,7 @@ function along with the right ride of the ODE and trial solution.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec98">Gradient Descent </h2>
<h2 id="___sec95">Gradient Descent </h2>
<p>
The idea of the gradient descent algorithm is to update parameters in
@@ -4117,7 +3986,7 @@ the elements in \( \hat{\omega} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec99">More on GD and cost function </h2>
<h2 id="___sec96">More on GD and cost function </h2>
<p>
In our case, we have to minimize the cost function \( c(x, P) \) with
@@ -4130,10 +3999,12 @@ This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
updated by
$$
\begin{aligned}
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
\begin{align}
P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P)
\label{_auto13}\\
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
\end{aligned}
\label{_auto14}
\end{align}
$$
<p>
@@ -4180,7 +4051,7 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec100">An implementation of a Deep Neural Network </h2>
<h2 id="___sec97">An implementation of a Deep Neural Network </h2>
<p>
As previously stated, a Deep Neural Network (DNN) follows the same
@@ -4196,54 +4067,7 @@ P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec101">Feed forward again </h2>
<p>
The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer.
<p>
The \( i \)-th neuron at layer \( l \) recieves the result
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
\( b_i^{(l), \ \mathrm{hidden} } \):
$$
\begin{aligned}
z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
&=
\begin{pmatrix}
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 \\
\hat{x}_j^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
$$
<p>
The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
$$
\begin{aligned}
\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
&=
\begin{pmatrix}
b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
\end{pmatrix}
\begin{pmatrix}
1 & 1 & \dots & 1 \\
\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
\end{pmatrix}
\end{aligned}
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec102">The final parts of the code </h2>
<h2 id="___sec98">The final parts of the code </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -4293,7 +4117,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec103">And adding Back propagation </h2>
<h2 id="___sec99">And adding Back propagation </h2>
<p>
This step is very similar for the neural network. The idea in this
@@ -4371,7 +4195,7 @@ analytically since Autograd does the work for us.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec104">Solving the ODE </h2>
<h2 id="___sec100">Solving the ODE </h2>
<p>
Finally, having set up the networks we are ready to use them to solve the ODE problem.
@@ -4386,14 +4210,14 @@ We add the analytical solution
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec105">Using neural network </h2>
<h2 id="___sec101">Using neural network </h2>
<p>
The code below solves the ODE using a neural network. The number of
values for the input \( \vec x \) is 10, number of hidden neurons in the
hidden layer being 10 and th step size used in gradien descent
\( \lambda = 0.001 \). The program updates the weights and biases in the
network <em>num_iter</em> times. Finally, it plots the results from using the
network for a given number of iterations. Finally, it plots the results from using the
neural network along with the analytical solution.
<p>
@@ -4430,7 +4254,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec106">Using a deep neural network </h2>
<h2 id="___sec102">Using a deep neural network </h2>
<p>
@@ -4463,7 +4287,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec107">Wrapping it up </h2>
<h2 id="___sec103">Wrapping it up </h2>
<p>
By rewriting the ODE as a minimization problem, it was possible to