@@ -410,7 +404,7 @@ network should find \( P \) such that it fulfills 102
-First, a feedforward of the inputs must be done. This means that \( \hat{x} \) -must be passed through an input layer, a hidden layer and a output - layer. The input layer in this case, does not need to process the - data any further. The input layer will consist of \( N_{\mathrm{input} } \) - neurons, passing its element to each neuron in the hidden layer. The - number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \). -
-For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \) -and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron -at the input layer is: + +
# Note that we use the numpy wrapper for Autograd (see the gradient descent slides)
+import autograd.numpy as np
+from autograd import grad, elementwise_grad
+import autograd.numpy.random as npr
+from matplotlib import pyplot as plt
-$$
-\begin{aligned}
-z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
-&=
-\begin{pmatrix}
-b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-x_j
-\end{pmatrix}
-\end{aligned}
-$$
+def sigmoid(z):
+ return 1/(1 + np.exp(-z))
+def neural_network(params, x):
+
+ # Find the weights (including and biases) for the hidden and output layer.
+ # Assume that params is a list of parameters for each layer.
+ # The biases are the first element for each array in params,
+ # and the weights are the remaning elements in each array in params.
+
+ w_hidden = params[0]
+ w_output = params[1]
+
+ # Assumes input x being an one-dimensional array
+ num_values = np.size(x)
+ x = x.reshape(-1, num_values)
+
+ # Assume that the input layer does nothing to the input x
+ x_input = x
+
+ ## Hidden layer:
+
+ # Add a row of ones to include bias
+ x_input = np.concatenate((np.ones((1,num_values)), x_input ), axis = 0)
+
+ z_hidden = np.matmul(w_hidden, x_input)
+ x_hidden = sigmoid(z_hidden)
+
+ ## Output layer:
+
+ # Include bias:
+ x_hidden = np.concatenate((np.ones((1,num_values)), x_hidden ), axis = 0)
+
+ z_output = np.matmul(w_output, x_hidden)
+ x_output = z_output
+
+ return x_output
+
@@ -418,7 +429,7 @@ $$
- + -
-The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector: +Now that the feedforward can be done, the next step is to decide how the +parameters should change such that they minimize the cost function. + +
+Recall that the chosen cost function for this problem is + $$ -\begin{aligned} -\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\ -&= -\begin{pmatrix} - b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}} -\end{pmatrix} -\begin{pmatrix} -1 & 1 & \dots & 1 \\ -x_1 & x_2 & \dots & x_N -\end{pmatrix} \\ -&= \hat{p}_{i, \mathrm{hidden}}^T X -\end{aligned} +c(x, P) = \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$
-It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row -in \( P_{\mathrm{hidden} } \), which contains the weights for the neural -network to minimize according to (19). +In order to minimize it, an optimalization method must be chosen.
-After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \) -in the hidden layer, the vector will be sent to an activation function -\( a_i(\hat{z}) \). In this example, the sigmoid function has been used: +Here, gradient descent with a constant step size has been chosen. -$$ -f(z) = \frac{1}{1 + \exp{(-z)}}. -$$ +
+Before looking at the gradient descent method, let us set up the cost +function along with the right ride of the ODE and trial solution. +
+ + +
# The trial solution using the deep neural network:
+def g_trial(x,params, g0 = 10):
+ return g0 + x*neural_network(params,x)
+
+# The right side of the ODE:
+def g(x, g_trial, gamma = 2):
+ return -gamma*g_trial
+
+# The cost function:
+def cost_function(P, x):
+
+ # Evaluate the trial function with the current parameters P
+ g_t = g_trial(x,P)
+
+ # Find the derivative w.r.t x of the neural network
+ d_net_out = elementwise_grad(neural_network,1)(P,x)
+
+ # Find the derivative w.r.t x of the trial function
+ d_g_t = elementwise_grad(g_trial,0)(x,P)
+
+ # The right side of the ODE
+ func = g(x, g_t)
+
+ err_sqr = (d_g_t - func)**2
+ cost_sum = np.sum(err_sqr)
+
+ return cost_sum
+
@@ -421,8 +434,6 @@ $$
-The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is: +The idea of the gradient descent algorithm is to update parameters in +direction where the cost function decreases goes to a minimum. + +
+In general, the update of some parameters \( \hat{\omega} \) given a cost +function defined by some weights \( \hat{\omega} \), \( c(x, \hat{\omega}) \), +goes as follows: $$ -\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big). +\hat{\omega}_{\mathrm{new} } = \hat{\omega} - \lambda \nabla_{\hat{\omega}} c(x, \hat{\omega}), $$
-The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer. +for a number of iterations or until \( \big|\big| \hat{\omega}_{\mathrm{new} } - \hat{\omega} \big|\big| \) +is smaller than some +given tolerance.
-The output layer consist of one neuron in this case, and combines the -output from each of the neurons in the hidden layers. The output layer -combines the results from the hidden layer using some weights \( -w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case, -it is assumes that the number of neurons in the output layer is one. - -
-The procedure of weigthing the output neuron \( j \) in the hidden layer -to the \( i \)-th neuron in the output layer is similar as for the hidden -layer described previously. - -$$ -\begin{aligned} -z_{1,j}^{\mathrm{output}} & = -\begin{pmatrix} -b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} -\end{pmatrix} -\begin{pmatrix} -1 \\ -\hat{x}_j^{\mathrm{hidden}} -\end{pmatrix} -\end{aligned} -$$ - -
-Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: - -$$ -\hat{z}_{1}^{\mathrm{output}} = -\begin{pmatrix} -b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}} -\end{pmatrix} -\begin{pmatrix} -1 & 1 & \dots & 1 \\ -\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}} -\end{pmatrix} -$$ - -
-In this case we seek a continous range of values since we are -approximating a function. This means that after computing -\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its -feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final -output of the network. +The value of \( \lambda \) decides how large steps the algorithm must take +in the direction of $ \nabla_{\hat{\omega}} c(x, \hat{\omega})$. The +notatation \( \nabla_{\hat{\omega}} \) denotes the gradient with respect to +the elements in \( \hat{\omega} \).
@@ -445,9 +405,6 @@ output of the network.
+In our case, we have to minimize the cost function \( c(x, P) \) with +respect to the two sets of weights and bisases, that is for the hidden +layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output} +} \) . + +
+This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is +updated by + +$$ +\begin{align} +P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) +\tag{20}\\ +P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P) +\tag{21} +\end{align} +$$ + +
+This might look like a cumberstone to set up the correct expression +for finding the gradients. Luckily, Autograd comes to the rescue. +
-
# Note that we use the numpy wrapper for Autograd (see the gradient descent slides)
-import autograd.numpy as np
-from autograd import grad, elementwise_grad
-import autograd.numpy.random as npr
-from matplotlib import pyplot as plt
-
-def sigmoid(z):
- return 1/(1 + np.exp(-z))
-
-def neural_network(params, x):
+def solve_ode_neural_network(x, num_neurons_hidden, num_iter, lmb):
+ ## Set up initial weigths and biases
- # Find the weights (including and biases) for the hidden and output layer.
- # Assume that params is a list of parameters for each layer.
- # The biases are the first element for each array in params,
- # and the weights are the remaning elements in each array in params.
-
- w_hidden = params[0]
- w_output = params[1]
+ # For the hidden layer
+ p0 = npr.randn(num_neurons_hidden, 2 )
- # Assumes input x being an one-dimensional array
- num_values = np.size(x)
- x = x.reshape(-1, num_values)
-
- # Assume that the input layer does nothing to the input x
- x_input = x
+ # For the output layer
+ p1 = npr.randn(1, num_neurons_hidden + 1 ) # +1 since bias is included
- ## Hidden layer:
-
- # Add a row of ones to include bias
- x_input = np.concatenate((np.ones((1,num_values)), x_input ), axis = 0)
-
- z_hidden = np.matmul(w_hidden, x_input)
- x_hidden = sigmoid(z_hidden)
+ P = [p0, p1]
- ## Output layer:
+ print('Initial cost: %g'%cost_function(P, x))
- # Include bias:
- x_hidden = np.concatenate((np.ones((1,num_values)), x_hidden ), axis = 0)
+ ## Start finding the optimal weigths using gradient descent
+
+ # Find the Python function that represents the gradient of the cost function
+ # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer
+ cost_function_grad = grad(cost_function,0)
+
+ # Let the update be done num_iter times
+ for i in range(num_iter):
+ # Evaluate the gradient at the current weights and biases in P.
+ # The cost_grad consist now of two arrays;
+ # one for the gradient w.r.t P_hidden and
+ # one for the gradient w.r.t P_output
+ cost_grad = cost_function_grad(P, x)
+
+ P[0] = P[0] - lmb * cost_grad[0]
+ P[1] = P[1] - lmb * cost_grad[1]
- z_output = np.matmul(w_output, x_hidden)
- x_output = z_output
-
- return x_output
+ print('Final cost: %g'%cost_function(P, x))
+
+ return P
@@ -434,10 +440,6 @@ MathJax.Hub.Config({
-Now that feedforward can be done, the next step is to decide how the -parameters should change such that they minimize the cost function. +As previously stated, a Deep Neural Network (DNN) follows the same +concept of a neural network, but having more than one hidden +layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers +where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The +input is still assumed to be an array of size \( 1 \times N \). The +network must now try to optimalize its output with respect to the +collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \ +P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \ +P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \). -
-Recall that the chosen cost function for this problem is - -$$ -c(x, P) = \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 -$$ - -
-In order to minimize it, an optimalization method must be chosen. - -
-Here, gradient descent with a constant step size has been chosen. - -
-Before looking at the gradient descent method, let us set up the cost -function along with the right ride of the ODE and trial solution. - -
- - -
# The trial solution using the deep neural network:
-def g_trial(x,params, g0 = 10):
- return g0 + x*neural_network(params,x)
-
-# The right side of the ODE:
-def g(x, g_trial, gamma = 2):
- return -gamma*g_trial
-
-# The cost function:
-def cost_function(P, x):
-
- # Evaluate the trial function with the current parameters P
- g_t = g_trial(x,P)
-
- # Find the derivative w.r.t x of the neural network
- d_net_out = elementwise_grad(neural_network,1)(P,x)
-
- # Find the derivative w.r.t x of the trial function
- d_g_t = elementwise_grad(g_trial,0)(x,P)
-
- # The right side of the ODE
- func = g(x, g_t)
-
- err_sqr = (d_g_t - func)**2
- cost_sum = np.sum(err_sqr)
-
- return cost_sum
-
@@ -439,11 +390,6 @@ function along with the right ride of the ODE and trial solution.
-The idea of the gradient descent algorithm is to update parameters in -direction where the cost function decreases goes to a minimum. -
-In general, the update of some parameters \( \hat{\omega} \) given a cost -function defined by some weights \( \hat{\omega} \), \( c(x, \hat{\omega}) \), -goes as follows: + +
def deep_neural_network(deep_params, x):
+ # N_hidden is the number of hidden layers
+ N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
+
+ # Assumes input x being an one-dimensional array
+ num_values = np.size(x)
+ x = x.reshape(-1, num_values)
+
+ # Assume that the input layer does nothing to the input x
+ x_input = x
+
+ # Due to multiple hidden layers, define a variable referencing to the
+ # output of the previous layer:
+ x_prev = x_input
+
+ ## Hidden layers:
+
+ for l in range(N_hidden):
+ # From the list of parameters P; find the correct weigths and bias for this layer
+ w_hidden = deep_params[l]
+
+ # Add a row of ones to include bias
+ x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)
-$$
-\hat{\omega}_{\mathrm{new} } = \hat{\omega} - \lambda \nabla_{\hat{\omega}} c(x, \hat{\omega}),
-$$
+ z_hidden = np.matmul(w_hidden, x_prev)
+ x_hidden = sigmoid(z_hidden)
-
-for a number of iterations or until \( \big|\big| \hat{\omega}_{\mathrm{new} } - \hat{\omega} \big|\big| \)
-is smaller than some
-given tolerance.
+ # Update x_prev such that next layer can use the output from this layer
+ x_prev = x_hidden
-
-The value of \( \lambda \) decides how large steps the algorithm must take
-in the direction of $ \nabla_{\hat{\omega}} c(x, \hat{\omega})$. The
-notatation \( \nabla_{\hat{\omega}} \) denotes the gradient with respect to
-the elements in \( \hat{\omega} \).
+ ## Output layer:
+
+ # Get the weights and bias for this layer
+ w_output = deep_params[-1]
+
+ # Include bias:
+ x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)
+ z_output = np.matmul(w_output, x_prev)
+ x_output = z_output
+
+ return x_output
+
@@ -410,10 +423,6 @@ the elements in \( \hat{\omega} \).
-In our case, we have to minimize the cost function \( c(x, P) \) with -respect to the two sets of weights and bisases, that is for the hidden -layer \( P_{\mathrm{hidden} } \) and for the ouput layer \( P_{\mathrm{output} -} \) . - -
-This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is -updated by - -$$ -\begin{aligned} -P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\ -P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P) -\end{aligned} -$$ - -
-This might look like a cumberstone to set up the correct expression -for finding the gradients. Luckily, Autograd comes to the rescue. +This step is very similar for the neural network. The idea in this +step is the same as for the neural network, but with more parameters +to update for. Again there is no need for computing the gradients +analytically since Autograd does the work for us.
-
def solve_ode_neural_network(x, num_neurons_hidden, num_iter, lmb):
+# The trial solution using the deep neural network:
+def g_trial_deep(x,params, g0 = 10):
+ return g0 + x*deep_neural_network(params,x)
+
+# The same cost function as for the neural network, but calls deep_neural_network instead.
+def cost_function_deep(P, x):
+
+ # Evaluate the trial function with the current parameters P
+ g_t = g_trial_deep(x,P)
+
+ # Find the derivative w.r.t x of the neural network
+ d_net_out = elementwise_grad(deep_neural_network,1)(P,x)
+
+ # Find the derivative w.r.t x of the trial function
+ d_g_t = elementwise_grad(g_trial_deep,0)(x,P)
+
+ # The right side of the ODE
+ func = g(x, g_t)
+
+ err_sqr = (d_g_t - func)**2
+ cost_sum = np.sum(err_sqr)
+
+ return cost_sum
+
+def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):
+ # num_hidden_neurons is now a list of number of neurons within each hidden layer
+
+ # Find the number of hidden layers:
+ N_hidden = np.size(num_neurons)
+
## Set up initial weigths and biases
- # For the hidden layer
- p0 = npr.randn(num_neurons_hidden, 2 )
+ # Initialize the list of parameters:
+ P = [None]*(N_hidden + 1) # + 1 to include the output layer
+ P[0] = npr.randn(num_neurons[0], 2 )
+ for l in range(1,N_hidden):
+ P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias
+
# For the output layer
- p1 = npr.randn(1, num_neurons_hidden + 1 ) # +1 since bias is included
+ P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included
- P = [p0, p1]
-
- print('Initial cost: %g'%cost_function(P, x))
+ print('Initial cost: %g'%cost_function_deep(P, x))
## Start finding the optimal weigths using gradient descent
# Find the Python function that represents the gradient of the cost function
# w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer
- cost_function_grad = grad(cost_function,0)
+ cost_function_deep_grad = grad(cost_function_deep,0)
# Let the update be done num_iter times
for i in range(num_iter):
# Evaluate the gradient at the current weights and biases in P.
- # The cost_grad consist now of two arrays;
- # one for the gradient w.r.t P_hidden and
- # one for the gradient w.r.t P_output
- cost_grad = cost_function_grad(P, x)
-
- P[0] = P[0] - lmb * cost_grad[0]
- P[1] = P[1] - lmb * cost_grad[1]
+ # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases
+ # in the hidden layers and output layers evaluated at x.
+ cost_deep_grad = cost_function_deep_grad(P, x)
+
+ for l in range(N_hidden+1):
+ P[l] = P[l] - lmb * cost_deep_grad[l]
- print('Final cost: %g'%cost_function(P, x))
+ print('Final cost: %g'%cost_function_deep(P, x))
return P
@@ -443,10 +450,6 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
-As previously stated, a Deep Neural Network (DNN) follows the same -concept of a neural network, but having more than one hidden -layer. Suppose that the network has \( N_{\mathrm{hidden}} \) hidden layers -where the \( l \)-th layer has \( N_{\mathrm{hidden}}^{(l)} \) neurons. The -input is still assumed to be an array of size \( 1 \times N \). The -network must now try to optimalize its output with respect to the -collection of weigths and biases \( P = \big\{P_{\mathrm{input} }, \ -P_{\mathrm{hidden} }^{(1)}, \ P_{\mathrm{hidden} }^{(2)}, \ \dots , \ -P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \). +Finally, having set up the networks we are ready to use them to solve the ODE problem. +We add the analytical solution +
+ + +
def g_analytic(x, gamma = 2, g0 = 10):
+ return g0*np.exp(-gamma*x)
+
@@ -395,10 +386,6 @@ P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
-The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer. +The code below solves the ODE using a neural network. The number of +values for the input \( \vec x \) is 10, number of hidden neurons in the +hidden layer being 10 and th step size used in gradien descent +\( \lambda = 0.001 \). The program updates the weights and biases in the +network for a given number of iterations. Finally, it plots the results from using the +neural network along with the analytical solution.
-The \( i \)-th neuron at layer \( l \) recieves the result -\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer -\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in -\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are -elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias -\( b_i^{(l), \ \mathrm{hidden} } \): -$$ -\begin{aligned} -z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\ -&= -\begin{pmatrix} -b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T -\end{pmatrix} -\begin{pmatrix} -1 \\ -\hat{x}_j^{(l-1),\mathrm{hidden} } -\end{pmatrix} -\end{aligned} -$$ + +
npr.seed(15)
-
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
+## Decide the vales of arguments to the function to solve
+N = 10
+x = np.linspace(0, 1, N)
-$$
-\begin{aligned}
-\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
-&=
-\begin{pmatrix}
-b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
-\end{pmatrix}
-\end{aligned}
-$$
+## Set up the initial parameters
+num_hidden_neurons = 10
+num_iter = 10000
+lmb = 0.001
+P = solve_ode_neural_network(x, num_hidden_neurons, num_iter, lmb)
+
+res = g_trial(x,P)
+res_analytical = g_analytic(x)
+
+print('Max absolute difference: %g'%np.max(np.abs(res - res_analytical)))
+
+plt.figure(figsize=(10,10))
+
+plt.title('Performance of neural network solving an ODE compared to the analytical solution')
+plt.plot(x, res_analytical)
+plt.plot(x, res[0,:])
+plt.legend(['analytical','nn'])
+plt.xlabel('x')
+plt.ylabel('g(x)')
+plt.show()
+
@@ -425,10 +414,6 @@ $$
- + + +
-
def deep_neural_network(deep_params, x):
- # N_hidden is the number of hidden layers
- N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
-
- # Assumes input x being an one-dimensional array
- num_values = np.size(x)
- x = x.reshape(-1, num_values)
-
- # Assume that the input layer does nothing to the input x
- x_input = x
-
- # Due to multiple hidden layers, define a variable referencing to the
- # output of the previous layer:
- x_prev = x_input
-
- ## Hidden layers:
-
- for l in range(N_hidden):
- # From the list of parameters P; find the correct weigths and bias for this layer
- w_hidden = deep_params[l]
-
- # Add a row of ones to include bias
- x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)
+npr.seed(15)
- z_hidden = np.matmul(w_hidden, x_prev)
- x_hidden = sigmoid(z_hidden)
+## Decide the vales of arguments to the function to solve
+N = 10
+x = np.linspace(0, 1, N)
- # Update x_prev such that next layer can use the output from this layer
- x_prev = x_hidden
+## Set up the initial parameters
+num_hidden_neurons = np.array([10,10])
+num_iter = 10000
+lmb = 0.001
- ## Output layer:
-
- # Get the weights and bias for this layer
- w_output = deep_params[-1]
-
- # Include bias:
- x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)
+P = solve_ode_deep_neural_network(x, num_hidden_neurons, num_iter, lmb)
- z_output = np.matmul(w_output, x_prev)
- x_output = z_output
+res = g_trial_deep(x,P)
+res_analytical = g_analytic(x)
- return x_output
+plt.figure(figsize=(10,10))
+
+plt.title('Performance of a deep neural network solving an ODE compared to the analytical solution')
+plt.plot(x, res_analytical)
+plt.plot(x, res[0,:])
+plt.legend(['analytical','dnn'])
+plt.ylabel('g(x)')
+plt.show()
@@ -427,10 +402,6 @@ MathJax.Hub.Config({
-This step is very similar for the neural network. The idea in this -step is the same as for the neural network, but with more parameters -to update for. Again there is no need for computing the gradients -analytically since Autograd does the work for us. +By rewriting the ODE as a minimization problem, it was possible to +solve equation using either a neural network (one hidden layer) or a +deep neural network (more than one hidden layers). How well the +network performed is measured by a specified cost function, which is +the function the network tries to minimize. Using a trial solution +which satisfies the additional condition and being defined by using +the output from the network in some way, the minimization problem +could be explicitly defined for out network to solve. The proposed +solution from the network is then the trial solution with parameters, +that is weights and biases within each layer in the network, such that +the solution minimizes the cost function. -
- - -
# The trial solution using the deep neural network:
-def g_trial_deep(x,params, g0 = 10):
- return g0 + x*deep_neural_network(params,x)
-
-# The same cost function as for the neural network, but calls deep_neural_network instead.
-def cost_function_deep(P, x):
-
- # Evaluate the trial function with the current parameters P
- g_t = g_trial_deep(x,P)
-
- # Find the derivative w.r.t x of the neural network
- d_net_out = elementwise_grad(deep_neural_network,1)(P,x)
-
- # Find the derivative w.r.t x of the trial function
- d_g_t = elementwise_grad(g_trial_deep,0)(x,P)
-
- # The right side of the ODE
- func = g(x, g_t)
-
- err_sqr = (d_g_t - func)**2
- cost_sum = np.sum(err_sqr)
-
- return cost_sum
-
-def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):
- # num_hidden_neurons is now a list of number of neurons within each hidden layer
-
- # Find the number of hidden layers:
- N_hidden = np.size(num_neurons)
-
- ## Set up initial weigths and biases
-
- # Initialize the list of parameters:
- P = [None]*(N_hidden + 1) # + 1 to include the output layer
-
- P[0] = npr.randn(num_neurons[0], 2 )
- for l in range(1,N_hidden):
- P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias
-
- # For the output layer
- P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included
-
- print('Initial cost: %g'%cost_function_deep(P, x))
-
- ## Start finding the optimal weigths using gradient descent
-
- # Find the Python function that represents the gradient of the cost function
- # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer
- cost_function_deep_grad = grad(cost_function_deep,0)
-
- # Let the update be done num_iter times
- for i in range(num_iter):
- # Evaluate the gradient at the current weights and biases in P.
- # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases
- # in the hidden layers and output layers evaluated at x.
- cost_deep_grad = cost_function_deep_grad(P, x)
-
- for l in range(N_hidden+1):
- P[l] = P[l] - lmb * cost_deep_grad[l]
-
- print('Final cost: %g'%cost_function_deep(P, x))
-
- return P
-
-First, a feedforward of the inputs must be done. This means that \( \hat{x} \) -must be passed through an input layer, a hidden layer and a output - layer. The input layer in this case, does not need to process the - data any further. The input layer will consist of \( N_{\mathrm{input} } \) - neurons, passing its element to each neuron in the hidden layer. The - number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \). - -
-For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \) -and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron -at the input layer is: - -
-$$
-\begin{aligned}
-z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
-&=
-\begin{pmatrix}
-b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-x_j
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector: -
-$$
-\begin{aligned}
-\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
-&=
-\begin{pmatrix}
- b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-x_1 & x_2 & \dots & x_N
-\end{pmatrix} \\
-&= \hat{p}_{i, \mathrm{hidden}}^T X
-\end{aligned}
-$$
-
-
-
-It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row -in \( P_{\mathrm{hidden} } \), which contains the weights for the neural -network to minimize according to (19). - -
-After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \) -in the hidden layer, the vector will be sent to an activation function -\( a_i(\hat{z}) \). In this example, the sigmoid function has been used: - -
-$$
-f(z) = \frac{1}{1 + \exp{(-z)}}.
-$$
-
-
-The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is: - -
-$$
-\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
-$$
-
-
-
-The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer. - -
-The output layer consist of one neuron in this case, and combines the -output from each of the neurons in the hidden layers. The output layer -combines the results from the hidden layer using some weights \( -w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case, -it is assumes that the number of neurons in the output layer is one. - -
-The procedure of weigthing the output neuron \( j \) in the hidden layer -to the \( i \)-th neuron in the output layer is similar as for the hidden -layer described previously. - -
-$$
-\begin{aligned}
-z_{1,j}^{\mathrm{output}} & =
-\begin{pmatrix}
-b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-\hat{x}_j^{\mathrm{hidden}}
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-
-Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer: - -
-$$
-\hat{z}_{1}^{\mathrm{output}} =
-\begin{pmatrix}
-b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
-\end{pmatrix}
-$$
-
-
-
-In this case we seek a continous range of values since we are -approximating a function. This means that after computing -\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its -feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final -output of the network. -
@@ -4171,10 +4032,10 @@ output of the network.
-Now that feedforward can be done, the next step is to decide how the
+Now that the feedforward can be done, the next step is to decide how the
parameters should change such that they minimize the cost function.
@@ -4231,7 +4092,7 @@ function along with the right ride of the ODE and trial solution.
The idea of the gradient descent algorithm is to update parameters in
@@ -4262,7 +4123,7 @@ the elements in \( \hat{\omega} \).
In our case, we have to minimize the cost function \( c(x, P) \) with
@@ -4276,10 +4137,12 @@ updated by
As previously stated, a Deep Neural Network (DNN) follows the same
@@ -4344,58 +4207,7 @@ P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
-The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer.
-
-
-The \( i \)-th neuron at layer \( l \) recieves the result
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
-\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are
-elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
-\( b_i^{(l), \ \mathrm{hidden} } \):
-
-
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
-
-
@@ -4446,7 +4258,7 @@ $$
This step is very similar for the neural network. The idea in this
@@ -4525,7 +4337,7 @@ analytically since Autograd does the work for us.
Finally, having set up the networks we are ready to use them to solve the ODE problem.
@@ -4541,14 +4353,14 @@ We add the analytical solution
The code below solves the ODE using a neural network. The number of
values for the input \( \vec x \) is 10, number of hidden neurons in the
hidden layer being 10 and th step size used in gradien descent
\( \lambda = 0.001 \). The program updates the weights and biases in the
-network num_iter times. Finally, it plots the results from using the
+network for a given number of iterations. Finally, it plots the results from using the
neural network along with the analytical solution.
@@ -4586,7 +4398,7 @@ plt.show()
@@ -4620,7 +4432,7 @@ plt.show()
By rewriting the ODE as a minimization problem, it was possible to
diff --git a/doc/pub/NeuralNet/html/NeuralNet-solarized.html b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
index 419531dea..f5c9407fe 100644
--- a/doc/pub/NeuralNet/html/NeuralNet-solarized.html
+++ b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
@@ -205,24 +205,20 @@ div { text-align: justify; text-justify: inter-word; }
('Reformulating the problem', 2, None, '___sec90'),
('Estimating errors', 2, None, '___sec91'),
('Creating a simple Deep Neural Net', 2, None, '___sec92'),
- ('Feedforward', 2, None, '___sec93'),
- ('Result after weighting', 2, None, '___sec94'),
- ('Output', 2, None, '___sec95'),
- ('Setting up the code, feed forward part', 2, None, '___sec96'),
- ('Backpropagation', 2, None, '___sec97'),
- ('Gradient Descent', 2, None, '___sec98'),
- ('More on GD and cost function', 2, None, '___sec99'),
+ ('Setting up the code, feed forward part', 2, None, '___sec93'),
+ ('Backpropagation', 2, None, '___sec94'),
+ ('Gradient Descent', 2, None, '___sec95'),
+ ('More on GD and cost function', 2, None, '___sec96'),
('An implementation of a Deep Neural Network',
2,
None,
- '___sec100'),
- ('Feed forward again', 2, None, '___sec101'),
- ('The final parts of the code', 2, None, '___sec102'),
- ('And adding Back propagation', 2, None, '___sec103'),
- ('Solving the ODE', 2, None, '___sec104'),
- ('Using neural network', 2, None, '___sec105'),
- ('Using a deep neural network', 2, None, '___sec106'),
- ('Wrapping it up', 2, None, '___sec107')]}
+ '___sec97'),
+ ('The final parts of the code', 2, None, '___sec98'),
+ ('And adding Back propagation', 2, None, '___sec99'),
+ ('Solving the ODE', 2, None, '___sec100'),
+ ('Using neural network', 2, None, '___sec101'),
+ ('Using a deep neural network', 2, None, '___sec102'),
+ ('Wrapping it up', 2, None, '___sec103')]}
end of tocinfo -->
-First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
-must be passed through an input layer, a hidden layer and a output
- layer. The input layer in this case, does not need to process the
- data any further. The input layer will consist of \( N_{\mathrm{input} } \)
- neurons, passing its element to each neuron in the hidden layer. The
- number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
-
-
-For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
-and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
-at the input layer is:
-
-$$
-\begin{aligned}
-z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
-&=
-\begin{pmatrix}
-b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-x_j
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-
-
-
-The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector:
-$$
-\begin{aligned}
-\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
-&=
-\begin{pmatrix}
- b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-x_1 & x_2 & \dots & x_N
-\end{pmatrix} \\
-&= \hat{p}_{i, \mathrm{hidden}}^T X
-\end{aligned}
-$$
-
-
-It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
-in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
-network to minimize according to \eqref{eq:min}.
-
-
-After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
-in the hidden layer, the vector will be sent to an activation function
-\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
-
-$$
-f(z) = \frac{1}{1 + \exp{(-z)}}.
-$$
-
-
-
-The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
-
-$$
-\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
-$$
-
-
-The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
-
-
-The output layer consist of one neuron in this case, and combines the
-output from each of the neurons in the hidden layers. The output layer
-combines the results from the hidden layer using some weights \(
-w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
-it is assumes that the number of neurons in the output layer is one.
-
-
-The procedure of weigthing the output neuron \( j \) in the hidden layer
-to the \( i \)-th neuron in the output layer is similar as for the hidden
-layer described previously.
-
-$$
-\begin{aligned}
-z_{1,j}^{\mathrm{output}} & =
-\begin{pmatrix}
-b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-\hat{x}_j^{\mathrm{hidden}}
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
-
-$$
-\hat{z}_{1}^{\mathrm{output}} =
-\begin{pmatrix}
-b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
-\end{pmatrix}
-$$
-
-
-In this case we seek a continous range of values since we are
-approximating a function. This means that after computing
-\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
-feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
-output of the network.
-
-
-
@@ -4026,10 +3895,10 @@ output of the network.
-Now that feedforward can be done, the next step is to decide how the
+Now that the feedforward can be done, the next step is to decide how the
parameters should change such that they minimize the cost function.
@@ -4083,7 +3952,7 @@ function along with the right ride of the ODE and trial solution.
The idea of the gradient descent algorithm is to update parameters in
@@ -4112,7 +3981,7 @@ the elements in \( \hat{\omega} \).
In our case, we have to minimize the cost function \( c(x, P) \) with
@@ -4125,10 +3994,12 @@ This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
updated by
$$
-\begin{aligned}
-P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
+\begin{align}
+P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P)
+\label{_auto13}\\
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
-\end{aligned}
+\label{_auto14}
+\end{align}
$$
@@ -4175,7 +4046,7 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
As previously stated, a Deep Neural Network (DNN) follows the same
@@ -4191,54 +4062,7 @@ P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
-The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer.
-
-
-The \( i \)-th neuron at layer \( l \) recieves the result
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
-\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are
-elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
-\( b_i^{(l), \ \mathrm{hidden} } \):
-
-$$
-\begin{aligned}
-z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
-&=
-\begin{pmatrix}
-b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-\hat{x}_j^{(l-1),\mathrm{hidden} }
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
-
-$$
-\begin{aligned}
-\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
-&=
-\begin{pmatrix}
-b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-
@@ -4288,7 +4112,7 @@ $$
This step is very similar for the neural network. The idea in this
@@ -4366,7 +4190,7 @@ analytically since Autograd does the work for us.
Finally, having set up the networks we are ready to use them to solve the ODE problem.
@@ -4381,14 +4205,14 @@ We add the analytical solution
The code below solves the ODE using a neural network. The number of
values for the input \( \vec x \) is 10, number of hidden neurons in the
hidden layer being 10 and th step size used in gradien descent
\( \lambda = 0.001 \). The program updates the weights and biases in the
-network num_iter times. Finally, it plots the results from using the
+network for a given number of iterations. Finally, it plots the results from using the
neural network along with the analytical solution.
@@ -4425,7 +4249,7 @@ plt.show()
-
@@ -4458,7 +4282,7 @@ plt.show()
By rewriting the ODE as a minimization problem, it was possible to
diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html
index ff0d73330..67f62dbb7 100644
--- a/doc/pub/NeuralNet/html/NeuralNet.html
+++ b/doc/pub/NeuralNet/html/NeuralNet.html
@@ -210,24 +210,20 @@ div { text-align: justify; text-justify: inter-word; }
('Reformulating the problem', 2, None, '___sec90'),
('Estimating errors', 2, None, '___sec91'),
('Creating a simple Deep Neural Net', 2, None, '___sec92'),
- ('Feedforward', 2, None, '___sec93'),
- ('Result after weighting', 2, None, '___sec94'),
- ('Output', 2, None, '___sec95'),
- ('Setting up the code, feed forward part', 2, None, '___sec96'),
- ('Backpropagation', 2, None, '___sec97'),
- ('Gradient Descent', 2, None, '___sec98'),
- ('More on GD and cost function', 2, None, '___sec99'),
+ ('Setting up the code, feed forward part', 2, None, '___sec93'),
+ ('Backpropagation', 2, None, '___sec94'),
+ ('Gradient Descent', 2, None, '___sec95'),
+ ('More on GD and cost function', 2, None, '___sec96'),
('An implementation of a Deep Neural Network',
2,
None,
- '___sec100'),
- ('Feed forward again', 2, None, '___sec101'),
- ('The final parts of the code', 2, None, '___sec102'),
- ('And adding Back propagation', 2, None, '___sec103'),
- ('Solving the ODE', 2, None, '___sec104'),
- ('Using neural network', 2, None, '___sec105'),
- ('Using a deep neural network', 2, None, '___sec106'),
- ('Wrapping it up', 2, None, '___sec107')]}
+ '___sec97'),
+ ('The final parts of the code', 2, None, '___sec98'),
+ ('And adding Back propagation', 2, None, '___sec99'),
+ ('Solving the ODE', 2, None, '___sec100'),
+ ('Using neural network', 2, None, '___sec101'),
+ ('Using a deep neural network', 2, None, '___sec102'),
+ ('Wrapping it up', 2, None, '___sec103')]}
end of tocinfo -->
-First, a feedforward of the inputs must be done. This means that \( \hat{x} \)
-must be passed through an input layer, a hidden layer and a output
- layer. The input layer in this case, does not need to process the
- data any further. The input layer will consist of \( N_{\mathrm{input} } \)
- neurons, passing its element to each neuron in the hidden layer. The
- number of neurons in the hidden layer will be \( N_{\mathrm{hidden} } \).
-
-
-For the \( i \)-th in the hidden layer with weight \( w_i^{\mathrm{hidden} } \)
-and bias \( b_i^{\mathrm{hidden} } \), the weighting from the \( j \)-th neuron
-at the input layer is:
-
-$$
-\begin{aligned}
-z_{i,j}^{\mathrm{hidden}} &= b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_j \\
-&=
-\begin{pmatrix}
-b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-x_j
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-
-
-
-The result after weighting the input at the \( i \)-th hidden neuron can be written as a vector:
-$$
-\begin{aligned}
-\hat{z}_{i}^{\mathrm{hidden}} &= \Big( b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}}x_1 , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_2, \ \dots \, , \ b_i^{\mathrm{hidden}} + w_i^{\mathrm{hidden}} x_N\Big) \\
-&=
-\begin{pmatrix}
- b_i^{\mathrm{hidden}} & w_i^{\mathrm{hidden}}
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-x_1 & x_2 & \dots & x_N
-\end{pmatrix} \\
-&= \hat{p}_{i, \mathrm{hidden}}^T X
-\end{aligned}
-$$
-
-
-It is the vector \( \hat{p}_{i, \mathrm{hidden}}^T \) that defines each row
-in \( P_{\mathrm{hidden} } \), which contains the weights for the neural
-network to minimize according to \eqref{eq:min}.
-
-
-After having found \( \hat{z}_{i}^{\mathrm{hidden}} \) for every neuron \( i \)
-in the hidden layer, the vector will be sent to an activation function
-\( a_i(\hat{z}) \). In this example, the sigmoid function has been used:
-
-$$
-f(z) = \frac{1}{1 + \exp{(-z)}}.
-$$
-
-
-
-The output $\hat{x}_i^{\mathrm{hidden}}$from each \( i \)-th hidden neuron is:
-
-$$
-\hat{x}_i^{\mathrm{hidden} } = f\big( \hat{z}_{i}^{\mathrm{hidden}} \big).
-$$
-
-
-The outputs \( \hat{x}_i^{\mathrm{hidden} } \) are then sent to the output layer.
-
-
-The output layer consist of one neuron in this case, and combines the
-output from each of the neurons in the hidden layers. The output layer
-combines the results from the hidden layer using some weights \(
-w_i^{\mathrm{output}} \) and biases \( b_i^{\mathrm{output}} \). In this case,
-it is assumes that the number of neurons in the output layer is one.
-
-
-The procedure of weigthing the output neuron \( j \) in the hidden layer
-to the \( i \)-th neuron in the output layer is similar as for the hidden
-layer described previously.
-
-$$
-\begin{aligned}
-z_{1,j}^{\mathrm{output}} & =
-\begin{pmatrix}
-b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-\hat{x}_j^{\mathrm{hidden}}
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-Expressing \( z_{1,j}^{\mathrm{output}} \) as a vector gives the following procedure of weighting the inputs from the hidden layer:
-
-$$
-\hat{z}_{1}^{\mathrm{output}} =
-\begin{pmatrix}
-b_1^{\mathrm{output}} & \hat{w}_1^{\mathrm{output}}
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-\hat{x}_1^{\mathrm{hidden}} & \hat{x}_2^{\mathrm{hidden}} & \dots & \hat{x}_N^{\mathrm{hidden}}
-\end{pmatrix}
-$$
-
-
-In this case we seek a continous range of values since we are
-approximating a function. This means that after computing
-\( \hat{z}_{1}^{\mathrm{output}} \) the neural network has finished its
-feedforward step, and \( \hat{z}_{1}^{\mathrm{output}} \) is the final
-output of the network.
-
-
-
@@ -4031,10 +3900,10 @@ output of the network.
-Now that feedforward can be done, the next step is to decide how the
+Now that the feedforward can be done, the next step is to decide how the
parameters should change such that they minimize the cost function.
@@ -4088,7 +3957,7 @@ function along with the right ride of the ODE and trial solution.
The idea of the gradient descent algorithm is to update parameters in
@@ -4117,7 +3986,7 @@ the elements in \( \hat{\omega} \).
In our case, we have to minimize the cost function \( c(x, P) \) with
@@ -4130,10 +3999,12 @@ This means that \( P_{\mathrm{hidden} } \) and \( P_{\mathrm{output} } \) is
updated by
$$
-\begin{aligned}
-P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
+\begin{align}
+P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P)
+\label{_auto13}\\
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
-\end{aligned}
+\label{_auto14}
+\end{align}
$$
@@ -4180,7 +4051,7 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
As previously stated, a Deep Neural Network (DNN) follows the same
@@ -4196,54 +4067,7 @@ P_{\mathrm{hidden} }^{(N_{\mathrm{hidden}})}, \ P_{\mathrm{output} }\big\} \).
-The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer.
-
-
-The \( i \)-th neuron at layer \( l \) recieves the result
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
-\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are
-elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
-\( b_i^{(l), \ \mathrm{hidden} } \):
-
-$$
-\begin{aligned}
-z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
-&=
-\begin{pmatrix}
-b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-\hat{x}_j^{(l-1),\mathrm{hidden} }
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-The output from the \( i \)-th neuron at the hidden layer \( l \) becomes a vector \( \hat{z}_{i}^{(l),\ \mathrm{hidden}} \):
-
-$$
-\begin{aligned}
-\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
-&=
-\begin{pmatrix}
-b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-
@@ -4293,7 +4117,7 @@ $$
This step is very similar for the neural network. The idea in this
@@ -4371,7 +4195,7 @@ analytically since Autograd does the work for us.
Finally, having set up the networks we are ready to use them to solve the ODE problem.
@@ -4386,14 +4210,14 @@ We add the analytical solution
The code below solves the ODE using a neural network. The number of
values for the input \( \vec x \) is 10, number of hidden neurons in the
hidden layer being 10 and th step size used in gradien descent
\( \lambda = 0.001 \). The program updates the weights and biases in the
-network num_iter times. Finally, it plots the results from using the
+network for a given number of iterations. Finally, it plots the results from using the
neural network along with the analytical solution.
@@ -4430,7 +4254,7 @@ plt.show()
-
@@ -4463,7 +4287,7 @@ plt.show()
By rewriting the ODE as a minimization problem, it was possible to
diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
index dad165cbf..4a8251538 100644
--- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
+++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
@@ -4266,148 +4266,9 @@
"For simplicity, we assume that the input is an array \n",
"$\\hat{x}= (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural\n",
"network should find $P$ such that it fulfills ([eq:min](#eq:min)).\n",
+ "All the ingredients discussed earlier, from the activation function, hidden layers and their weights, biases etc\n",
+ "are included below.\n",
"\n",
- "## Feedforward\n",
- "\n",
- "First, a feedforward of the inputs must be done. This means that $\\hat{x}$ \n",
- "must be passed through an input layer, a hidden layer and a output\n",
- " layer. The input layer in this case, does not need to process the\n",
- " data any further. The input layer will consist of $N_{\\mathrm{input} }$\n",
- " neurons, passing its element to each neuron in the hidden layer. The\n",
- " number of neurons in the hidden layer will be $N_{\\mathrm{hidden} }$.\n",
- "\n",
- "For the $i$-th in the hidden layer with weight $w_i^{\\mathrm{hidden} }$\n",
- "and bias $b_i^{\\mathrm{hidden} }$, the weighting from the $j$-th neuron\n",
- "at the input layer is:"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "$$\n",
- "\\begin{aligned}\n",
- "z_{i,j}^{\\mathrm{hidden}} &= b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}}x_j \\\\\n",
- "&= \n",
- "\\begin{pmatrix}\n",
- "b_i^{\\mathrm{hidden}} & w_i^{\\mathrm{hidden}}\n",
- "\\end{pmatrix}\n",
- "\\begin{pmatrix}\n",
- "1 \\\\\n",
- "x_j\n",
- "\\end{pmatrix} \n",
- "\\end{aligned}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "## Result after weighting\n",
- "\n",
- "The result after weighting the input at the $i$-th hidden neuron can be written as a vector:"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "$$\n",
- "\\begin{aligned}\n",
- "\\hat{z}_{i}^{\\mathrm{hidden}} &= \\Big( b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}}x_1 , \\ b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}} x_2, \\ \\dots \\, , \\ b_i^{\\mathrm{hidden}} + w_i^{\\mathrm{hidden}} x_N\\Big) \\\\\n",
- "&= \n",
- "\\begin{pmatrix}\n",
- " b_i^{\\mathrm{hidden}} & w_i^{\\mathrm{hidden}}\n",
- "\\end{pmatrix}\n",
- "\\begin{pmatrix}\n",
- "1 & 1 & \\dots & 1 \\\\\n",
- "x_1 & x_2 & \\dots & x_N\n",
- "\\end{pmatrix} \\\\\n",
- "&= \\hat{p}_{i, \\mathrm{hidden}}^T X\n",
- "\\end{aligned}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "It is the vector $\\hat{p}_{i, \\mathrm{hidden}}^T$ that defines each row\n",
- "in $P_{\\mathrm{hidden} }$, which contains the weights for the neural\n",
- "network to minimize according to ([eq:min](#eq:min)).\n",
- "\n",
- "After having found $\\hat{z}_{i}^{\\mathrm{hidden}} $ for every neuron $i$\n",
- "in the hidden layer, the vector will be sent to an activation function\n",
- "$a_i(\\hat{z})$. In this example, the sigmoid function has been used:\n",
- "\n",
- "$$\n",
- "f(z) = \\frac{1}{1 + \\exp{(-z)}}.\n",
- "$$\n",
- "\n",
- "\n",
- "## Output\n",
- "\n",
- "The output $\\hat{x}_i^{\\mathrm{hidden}}$from each $i$-th hidden neuron is:\n",
- "\n",
- "$$\n",
- "\\hat{x}_i^{\\mathrm{hidden} } = f\\big( \\hat{z}_{i}^{\\mathrm{hidden}} \\big).\n",
- "$$\n",
- "\n",
- "The outputs $\\hat{x}_i^{\\mathrm{hidden} } $ are then sent to the output layer. \n",
- "\n",
- "The output layer consist of one neuron in this case, and combines the\n",
- "output from each of the neurons in the hidden layers. The output layer\n",
- "combines the results from the hidden layer using some weights $\n",
- "w_i^{\\mathrm{output}}$ and biases $b_i^{\\mathrm{output}}$. In this case,\n",
- "it is assumes that the number of neurons in the output layer is one.\n",
- "\n",
- "The procedure of weigthing the output neuron $j$ in the hidden layer\n",
- "to the $i$-th neuron in the output layer is similar as for the hidden\n",
- "layer described previously."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "$$\n",
- "\\begin{aligned}\n",
- "z_{1,j}^{\\mathrm{output}} & = \n",
- "\\begin{pmatrix}\n",
- "b_1^{\\mathrm{output}} & \\hat{w}_1^{\\mathrm{output}}\n",
- "\\end{pmatrix}\n",
- "\\begin{pmatrix}\n",
- "1 \\\\\n",
- "\\hat{x}_j^{\\mathrm{hidden}}\n",
- "\\end{pmatrix}\n",
- "\\end{aligned}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Expressing $z_{1,j}^{\\mathrm{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:\n",
- "\n",
- "$$\n",
- "\\hat{z}_{1}^{\\mathrm{output}} = \n",
- "\\begin{pmatrix}\n",
- "b_1^{\\mathrm{output}} & \\hat{w}_1^{\\mathrm{output}}\n",
- "\\end{pmatrix}\n",
- "\\begin{pmatrix}\n",
- "1 & 1 & \\dots & 1 \\\\\n",
- "\\hat{x}_1^{\\mathrm{hidden}} & \\hat{x}_2^{\\mathrm{hidden}} & \\dots & \\hat{x}_N^{\\mathrm{hidden}}\n",
- "\\end{pmatrix}\n",
- "$$\n",
- "\n",
- "In this case we seek a continous range of values since we are\n",
- "approximating a function. This means that after computing\n",
- "$\\hat{z}_{1}^{\\mathrm{output}}$ the neural network has finished its\n",
- "feedforward step, and $\\hat{z}_{1}^{\\mathrm{output}}$ is the final\n",
- "output of the network.\n",
"\n",
"\n",
"## Setting up the code, feed forward part"
@@ -4472,7 +4333,7 @@
"source": [
"## Backpropagation\n",
"\n",
- "Now that feedforward can be done, the next step is to decide how the\n",
+ "Now that the feedforward can be done, the next step is to decide how the\n",
"parameters should change such that they minimize the cost function.\n",
"\n",
"Recall that the chosen cost function for this problem is"
@@ -4588,11 +4449,29 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "\n",
+ "Backpropagation
+Backpropagation
Gradient Descent
+Gradient Descent
More on GD and cost function
+More on GD and cost function
$$
-\begin{aligned}
-P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P) \\
+\begin{align}
+P_{\mathrm{hidden},\mathrm{new}} &= P_{\mathrm{hidden}} - \lambda \nabla_{P_{\mathrm{hidden}}} c(x, P)
+\tag{20}\\
P_{\mathrm{output},\mathrm{new}} &= P_{\mathrm{output}} - \lambda \nabla_{P_{\mathrm{output}}} c(x, P)
-\end{aligned}
+\tag{21}
+\end{align}
$$
@@ -4328,7 +4191,7 @@ for finding the gradients. Luckily, Autograd comes to the rescue.
An implementation of a Deep Neural Network
+An implementation of a Deep Neural Network
Feed forward again
-
-
-$$
-\begin{aligned}
-z_{i,j}^{(l),\ \mathrm{hidden}} &= b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_j^{(l-1),\mathrm{hidden} } \\
-&=
-\begin{pmatrix}
-b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
-\end{pmatrix}
-\begin{pmatrix}
-1 \\
-\hat{x}_j^{(l-1),\mathrm{hidden} }
-\end{pmatrix}
-\end{aligned}
-$$
-
-
-
-$$
-\begin{aligned}
-\hat{z}_{i}^{(l),\ \mathrm{hidden}} &= \Big( b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_1^{(l-1),\mathrm{hidden} }, \ \dots \ , \ b_i^{(l), \ \mathrm{hidden}} + \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T\hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} } \Big) \\
-&=
-\begin{pmatrix}
-b_i^{(l), \ \mathrm{hidden}} & \big(\hat{w}_{i}^{(l), \ \mathrm{hidden}}\big)^T
-\end{pmatrix}
-\begin{pmatrix}
-1 & 1 & \dots & 1 \\
-\hat{x}_{1}^{(l-1),\mathrm{hidden} } & \hat{x}_{2}^{(l-1),\mathrm{hidden} } & \dots & \hat{x}_{N_{hidden}^{(l-1)}}^{(l-1),\mathrm{hidden} }
-\end{pmatrix}
-\end{aligned}
-$$
-
-The final parts of the code
+The final parts of the code
And adding Back propagation
+And adding Back propagation
Solving the ODE
+Solving the ODE
Using neural network
+Using neural network
Using a deep neural network
+Using a deep neural network
Wrapping it up
+Wrapping it up
-Feedforward
-
-Result after weighting
-
-
-
-Output
-
-
-
-Setting up the code, feed forward part
+Setting up the code, feed forward part
-Backpropagation
+Backpropagation
-Gradient Descent
+Gradient Descent
-More on GD and cost function
+More on GD and cost function
-An implementation of a Deep Neural Network
+An implementation of a Deep Neural Network
-Feed forward again
-
-
-
-The final parts of the code
+The final parts of the code
-And adding Back propagation
+And adding Back propagation
-Solving the ODE
+Solving the ODE
-Using neural network
+Using neural network
Using a deep neural network
+Using a deep neural network
-Wrapping it up
+Wrapping it up
-Feedforward
-
-Result after weighting
-
-
-
-Output
-
-
-
-Setting up the code, feed forward part
+Setting up the code, feed forward part
-Backpropagation
+Backpropagation
-Gradient Descent
+Gradient Descent
-More on GD and cost function
+More on GD and cost function
-An implementation of a Deep Neural Network
+An implementation of a Deep Neural Network
-Feed forward again
-
-
-
-The final parts of the code
+The final parts of the code
-And adding Back propagation
+And adding Back propagation
-Solving the ODE
+Solving the ODE
-Using neural network
+Using neural network
Using a deep neural network
+Using a deep neural network
-Wrapping it up
+Wrapping it up