Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories. For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer: $$ z_{j}^{h} = \sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \boldsymbol{x}^T \boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$
this is then passed through our activation function $$ a_{j}^{h} = f(z_{j}^{h}) .$$
We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron \( j \) in the output layer: $$ z_{j}^{o} = \sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\boldsymbol{a}^{h})^T \boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$
Finally we calculate the output of neuron \( j \) in the output layer using the softmax function: $$ a_{j}^{o} = \frac{\exp{(z_j^{o})}} {\sum_{c=0}^{C-1} \exp{(z_c^{o})}} .$$