The next step is to decide how the neural net \( N(x, P) \) should be. In this case, the neural network is made from scratch to understand better how a neural network works, gain more control over its architecture, and see how Autograd can be used to simplify the implementation.
Since a deep neural network (DNN) is a neural network with more than one hidden layer, we can first look on how to implement a neural network. Having an implementation of a neural network at hand, an extension of it into a deep neural network would (hopefully) be painless.
For simplicity, we assume that the input is an array \( \hat{x}= (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (19). All the ingredients discussed earlier, from the activation function, hidden layers and their weights, biases etc are included below.