diff --git a/doc/src/NeuralNet/diffeq.ipynb b/doc/src/NeuralNet/diffeq.ipynb new file mode 100644 index 000000000..7661dc174 --- /dev/null +++ b/doc/src/NeuralNet/diffeq.ipynb @@ -0,0 +1,737 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "%matplotlib inline" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Example : Exponential decay in one dimension \n", + "\n", + "In this notebook we will see how a neural network performs when solving the equation \n", + "\n", + "$$\n", + "\\label{eq:ode}\n", + "g'(x) = -\\gamma g(x) \n", + "$$\n", + "\n", + "where $g(0) = g_0$ with $\\gamma$ and $g_0$ being some chosen values. This equation is an ordinary differential equation since the function we have to solve for, $g(x)$, is of one variable.\n", + "\n", + "In this example, $\\gamma = 2$ and $g_0 = 10$ but feel free to change them and see how the neural network performs.\n", + "\n", + "## Trial solution\n", + "\n", + "To begin with, a trial solution $g_t(t)$ must be chosen. A general trial solution for ordinary differential equations could be\n", + "\n", + "$$\n", + "g_t(x, P) = h_1(x) + h_2(x, N(x, P))\n", + "$$\n", + "\n", + "with $h_1(x)$ ensuring that $g_t(x)$ satisfies some conditions and $h_2(x,N(x, P))$ an expression involving $x$ and the output from the neural network $N(x,P)$ with $P $ being the collection of the weights and biases for each layer. It is assumed that there are no weights and bias at the input layer, so $P = \\{ P_{\\text{hidden}}, P_{\\text{output}} \\}$. If there are $N_{\\text{hidden} }$ neurons in the hidden layer, then $P_{\\text{hidden}}$ is a $N_{\\text{hidden} } \\times 2$ matrix. The first column in $P_{\\text{hidden} }$ represents the bias for each neuron in the hidden layer and the second column represents the weigths for each neuron. If there are $N_{\\text{output} }$ neurons in the output layer, then $P_{\\text{output}} $ is a $N_{\\text{output} } \\times (1 + N_{\\text{hidden} })$ matrix. Its first column represents the bias of each neuron and the remaining columns represents the weights to each neuron. \n", + "\n", + "It is given that $g(0) = g_0$. The trial solution must fulfill this condition to be a proper solution of \\eqref{eq:ode}. A possible way to ensure that $g_t(0, P) = g_0$, is to let $F(N(x,P)) = x\\cdot N(x,P)$ and $A(x) = g_0$. This gives the following trial solution:\n", + "\n", + "$$\n", + "\\label{eq:trial}\n", + "g_t(x, P) = g_0 + x \\cdot N(x, P)\n", + "$$\n", + "\n", + "\n", + "\n", + "## Reformulating the problem \n", + "\n", + "Often, the role of a neural network is to minimize its parameters with respect to some given error criteria. This criteria, the cost or loss function, is a measure of how much error the output of the network has compared to some given known answers. A reformulation of \\eqref{eq:ode} must therefore be done, such that it describes the problem a neural network can solve. \n", + "\n", + "The neural network must find the set of weigths and biases $P$ such that the trial solution in \\eqref{eq:trial} satisfies \\eqref{eq:ode}. The trial solution has been chosen such that it already solves the condition $g(0) = g_0$. What remains, is to find $P$ such that \n", + "\n", + "$$ \n", + "\\label{eq:nnmin}\n", + "g_t'(x, P) = - \\gamma g_t(x, P) \n", + "$$\n", + "\n", + "is fulfilled as *best as possible*. The left hand and right side of \\eqref{eq:nnmin} must be computed seperately, and then the neural network will choose which weights and biases in $P$ makes the sides as equal as possible. Having two sides of an equation as equal as possible, means that the absolute or squared difference between the sides must be as close to zero as small. In this case, the difference squared is an appropiate measurement of how errorneous the trial solution is with respect to $P$ of the neural network. Therefore, the problem our network must solve, is\n", + "\n", + "$$\n", + "\\min_{P}\\Big\\{ \\big(g_t'(x, P) - ( -\\gamma g_t(x, P) \\big)^2 \\Big\\}\n", + "$$\n", + "\n", + "or, in terms of weights and biases for each layer:\n", + "\n", + "$$\n", + "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) \\big)^2 \\Big\\}\n", + "$$\n", + "\n", + "for an input value $x$. \n", + "If the neural network evaluates $g_t(x, P)$ at more avalues for $x$, say $N$ values $x_i$ for $i = 1, \\dots, N$, then the *total* error to minimize is \n", + "\n", + "$$ \\label{eq:min}\n", + "\\min_{P}\\Big\\{\\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2 \\Big\\} \n", + "$$\n", + "\n", + "Letting $c(x, P) = \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem of which our network must solve, is\n", + "\n", + "$$\n", + "\\min_{P} c(x, P)\n", + "$$\n", + "\n", + "or in terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$\n", + "\n", + "$$\n", + "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(x, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n", + "$$\n", + "\n", + "## Creating a simple Deep Neural Net\n", + "\n", + "The next step is to decide how the neural net $N(x, P)$ in \\eqref{eq:trial} should be. In this case, the neural network is made from scratch to understand better how a neural network works, gain more control over its architecture, and see how Autograd can be used to simplify the implementation. \n", + "\n", + "### An implementation of a Neural Network\n", + "\n", + "Since a deep neural network (DNN) is a neural network with more than one hidden layer, we can first look on how to implement a neural network. Having an implementation of a neural network at hand, an extension of it into a deep neural network would (hopefully) be painless. \n", + "\n", + "For simplicity, it is assumed that the input is an array $\\vec x = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills \\eqref{eq:min}. \n", + "\n", + "#### Feedforward\n", + "First, a feedforward of the inputs must be done. This means that $\\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of $N_{\\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\\text{hidden} }$. \n", + "\n", + "For the $i$-th in the hidden layer with weight $w_i^{\\text{hidden} }$ and bias $b_i^{\\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is:\n", + "\n", + "$$ \n", + "\\begin{aligned}\n", + "z_{i,j}^{\\text{hidden}} &= b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_j \\\\\n", + "&= \n", + "\\begin{pmatrix}\n", + "b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n", + "\\end{pmatrix}\n", + "\\begin{pmatrix}\n", + "1 \\\\\n", + "x_j\n", + "\\end{pmatrix} \n", + "\\end{aligned}\n", + "$$\n", + "\n", + "The result after weighting the input at the $i$-th hidden neuron can be written as a vector:\n", + "\n", + "$$\n", + "\\begin{aligned}\n", + "\\vec{z}_{i}^{\\text{hidden}} &= \\Big( b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_1 , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_2, \\ \\dots \\, , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_N\\Big) \\\\\n", + "&= \n", + "\\begin{pmatrix}\n", + " b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n", + "\\end{pmatrix}\n", + "\\begin{pmatrix}\n", + "1 & 1 & \\dots & 1 \\\\\n", + "x_1 & x_2 & \\dots & x_N\n", + "\\end{pmatrix} \\\\\n", + "&= \\vec{p}_{i, \\text{hidden}}^T X\n", + "\\end{aligned}\n", + "$$\n", + "\n", + "It is the vector $\\vec{p}_{i, \\text{hidden}}^T$ that defines each row in $P_{\\text{hidden} }$, which contains the weights for the neural network to minimize according to \\eqref{eq:min}. \n", + "\n", + "After having found $\\vec{z}_{i}^{\\text{hidden}} $ for every neuron $i$ in the hidden layer, the vector will be sent to an activation function $a_i(\\vec{z})$. \n", + "In this example, the sigmoid function has been used:\n", + "\n", + "$$\n", + "f(z) = \\frac{1}{1 + \\exp{(-z)}}\n", + "$$\n", + "\n", + "but feel free to chose any activation function you like. \n", + "\n", + "The output $\\vec{x}_i^{\\text{hidden} }$from each $i$-th hidden neuron is:\n", + "\n", + "$$\n", + "\\vec{x}_i^{\\text{hidden} } = f\\big( \\vec{z}_{i}^{\\text{hidden}} \\big)\n", + "$$\n", + "\n", + "The outputs $\\vec{x}_i^{\\text{hidden} } $ are then sent to the output layer. \n", + "\n", + "The output layer consist of one neuron in this case, and combines the output from each of the neurons in the hidden layers. The output layer combines the results from the hidden layer using some weights $ w_i^{\\text{output}}$ and biases $b_i^{\\text{output}}$. In this case, it is assumes that the number of neurons in the output layer is one. \n", + "\n", + "The procedure of weigthing the output neuron $j$ in the hidden layer to the $i$-th neuron in the output layer is similar as for the hidden layer described previously. \n", + "\n", + "$$\n", + "\\begin{aligned}\n", + "z_{1,j}^{\\text{output}} & = \n", + "\\begin{pmatrix}\n", + "b_1^{\\text{output}} & \\vec{w}_1^{\\text{output}}\n", + "\\end{pmatrix}\n", + "\\begin{pmatrix}\n", + "1 \\\\\n", + "\\vec{x}_j^{\\text{hidden}}\n", + "\\end{pmatrix}\n", + "\\end{aligned}\n", + "$$\n", + "\n", + "Expressing $z_{1,j}^{\\text{output}}$ as a vector gives the following procedure of weighting the inputs from the hidden layer:\n", + "\n", + "$$\n", + "\\vec{z}_{1}^{\\text{output}} = \n", + "\\begin{pmatrix}\n", + "b_1^{\\text{output}} & \\vec{w}_1^{\\text{output}}\n", + "\\end{pmatrix}\n", + "\\begin{pmatrix}\n", + "1 & 1 & \\dots & 1 \\\\\n", + "\\vec{x}_1^{\\text{hidden}} & \\vec{x}_2^{\\text{hidden}} & \\dots & \\vec{x}_N^{\\text{hidden}}\n", + "\\end{pmatrix}\n", + "$$\n", + "\n", + "In this case we seek a continous range of values since we are approximating a function. This means that after computing $\\vec{z}_{1}^{\\text{output}}$ the neural network has finished its feedforward step, and $\\vec{z}_{1}^{\\text{output}}$ is the final output of the network." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Autograd will be used for later, so the numpy wrapper for Autograd must be imported\n", + "import autograd.numpy as np\n", + "from autograd import grad, elementwise_grad\n", + "import autograd.numpy.random as npr\n", + "from matplotlib import pyplot as plt\n", + "\n", + "def sigmoid(z):\n", + " return 1/(1 + np.exp(-z))\n", + "\n", + "def neural_network(params, x):\n", + " \n", + " # Find the weights (including and biases) for the hidden and output layer.\n", + " # Assume that params is a list of parameters for each layer. \n", + " # The biases are the first element for each array in params, \n", + " # and the weights are the remaning elements in each array in params. \n", + " \n", + " w_hidden = params[0]\n", + " w_output = params[1]\n", + "\n", + " # Assumes input x being an one-dimensional array\n", + " num_values = np.size(x)\n", + " x = x.reshape(-1, num_values)\n", + " \n", + " # Assume that the input layer does nothing to the input x\n", + " x_input = x\n", + "\n", + " ## Hidden layer:\n", + " \n", + " # Add a row of ones to include bias\n", + " x_input = np.concatenate((np.ones((1,num_values)), x_input ), axis = 0)\n", + " \n", + " z_hidden = np.matmul(w_hidden, x_input)\n", + " x_hidden = sigmoid(z_hidden)\n", + "\n", + " ## Output layer:\n", + " \n", + " # Include bias:\n", + " x_hidden = np.concatenate((np.ones((1,num_values)), x_hidden ), axis = 0)\n", + "\n", + " z_output = np.matmul(w_output, x_hidden)\n", + " x_output = z_output\n", + "\n", + " return x_output" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Backpropagation\n", + "\n", + "Now that feedforward can be done, the next step is to decide how the parameters should change such that they minimize the cost function. \n", + "\n", + "Recall that the chosen cost function for this problem is \n", + "\n", + "$$\n", + "c(x, P) = \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n", + "$$\n", + "\n", + "In order to minimize it, an optimalization method must be chosen. \n", + "\n", + "Here, gradient descent with a constant step size has been chosen. \n", + "\n", + "Before looking at the gradient descent method, let us set up the cost function along with the right ride of the ODE and trial solution." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# The trial solution using the deep neural network:\n", + "def g_trial(x,params, g0 = 10):\n", + " return g0 + x*neural_network(params,x)\n", + "\n", + "# The right side of the ODE:\n", + "def g(x, g_trial, gamma = 2):\n", + " return -gamma*g_trial\n", + "\n", + "# The cost function:\n", + "def cost_function(P, x):\n", + " \n", + " # Evaluate the trial function with the current parameters P\n", + " g_t = g_trial(x,P)\n", + " \n", + " # Find the derivative w.r.t x of the neural network\n", + " d_net_out = elementwise_grad(neural_network,1)(P,x) \n", + " \n", + " # Find the derivative w.r.t x of the trial function\n", + " d_g_t = elementwise_grad(g_trial,0)(x,P) \n", + " \n", + " # The right side of the ODE \n", + " func = g(x, g_t)\n", + "\n", + " err_sqr = (d_g_t - func)**2\n", + " cost_sum = np.sum(err_sqr)\n", + " \n", + " return cost_sum" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### Gradient Descent\n", + "\n", + "The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum. \n", + "\n", + "In general, the update of some parameters $\\vec \\omega$ given a cost function defined by some weights $\\vec \\omega$, $c(x, \\vec \\omega)$, goes as follows:\n", + "\n", + "$$\n", + "\\vec \\omega_{\\text{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(x, \\vec \\omega)\n", + "$$\n", + "\n", + "for a number of iterations or until $ \\big|\\big| \\vec \\omega_{\\text{new} } - \\vec \\omega \\big|\\big|$ is smaller than some given tolerance. \n", + "\n", + "The value of $\\lambda$ decides how large steps the algorithm must take in the direction of $ \\nabla_{\\vec \\omega} c(x, \\vec \\omega)$. The notatation $\\nabla_{\\vec \\omega}$ denotes the gradient with respect to the elements in $\\vec \\omega$. \n", + "\n", + "In our case, we have to minimize the cost function $c(x, P)$ with respect to the two sets of weights and bisases, that is for the hidden layer $P_{\\text{hidden} }$ and for the ouput layer $P_{\\text{output} }$ .\n", + "\n", + "This means that $P_{\\text{hidden} }$ and $P_{\\text{output} }$ is updated by\n", + "\n", + "$$\n", + "\\begin{aligned}\n", + "P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(x, P) \\\\\n", + "P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(x, P) \n", + "\\end{aligned}\n", + "$$\n", + "\n", + "This might look like a cumberstone to set up the correct expression for finding the gradients. Luckily, Autograd comes to the rescue. " + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "def solve_ode_neural_network(x, num_neurons_hidden, num_iter, lmb):\n", + " ## Set up initial weigths and biases \n", + " \n", + " # For the hidden layer\n", + " p0 = npr.randn(num_neurons_hidden, 2 ) \n", + "\n", + " # For the output layer\n", + " p1 = npr.randn(1, num_neurons_hidden + 1 ) # +1 since bias is included\n", + "\n", + " P = [p0, p1]\n", + "\n", + " print('Initial cost: %g'%cost_function(P, x))\n", + " \n", + " ## Start finding the optimal weigths using gradient descent\n", + " \n", + " # Find the Python function that represents the gradient of the cost function\n", + " # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer\n", + " cost_function_grad = grad(cost_function,0)\n", + " \n", + " # Let the update be done num_iter times\n", + " for i in range(num_iter):\n", + " # Evaluate the gradient at the current weights and biases in P. \n", + " # The cost_grad consist now of two arrays; \n", + " # one for the gradient w.r.t P_hidden and \n", + " # one for the gradient w.r.t P_output\n", + " cost_grad = cost_function_grad(P, x)\n", + " \n", + " P[0] = P[0] - lmb * cost_grad[0]\n", + " P[1] = P[1] - lmb * cost_grad[1]\n", + "\n", + " print('Final cost: %g'%cost_function(P, x))\n", + " \n", + " return P" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### An implementation of a Deep Neural Network\n", + "\n", + "As previously stated, a Deep Neural Network (DNN) follows the same concept of a neural network, but having more than one hidden layer. Suppose that the network has $N_{\\text{hidden}}$ hidden layers where the $l$-th layer has $N_{\\text{hidden}}^{(l)}$ neurons. The input is still assumed to be an array of size $1 \\times N$. The network must now try to optimalize its output with respect to the collection of weigths and biases $P = \\big\\{P_{\\text{input} }, \\ P_{\\text{hidden} }^{(1)}, \\ P_{\\text{hidden} }^{(2)}, \\ \\dots , \\ P_{\\text{hidden} }^{(N_{\\text{hidden}})}, \\ P_{\\text{output} }\\big\\}$.\n", + "\n", + "#### Feedforward \n", + "\n", + "The feedforward step is similar to as for the neural netowork, but now considering more than one hidden layer. \n", + "\n", + "The $i$-th neuron at layer $l$ recieves the result $\\vec{x}_j^{(l-1),\\text{hidden} }$ from the $j$-th neuron at layer $l-1$. The $i$-th neuron at layer $l$ weights all of the elements in $\\vec{x}_j^{(l-1),\\text{hidden} }$ with a weight vector $\\vec w_{i,j}^{(l), \\ \\text{hidden} }$ with as many weigths as there are elements in$\\vec{x}_j^{(l-1),\\text{hidden} }$, and adds a bias $b_i^{(l), \\ \\text{hidden} }$:\n", + "\n", + "$$\n", + "\\begin{aligned}\n", + "z_{i,j}^{(l),\\ \\text{hidden}} &= b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_j^{(l-1),\\text{hidden} } \\\\\n", + "&= \n", + "\\begin{pmatrix}\n", + "b_i^{(l), \\ \\text{hidden}} & \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\n", + "\\end{pmatrix}\n", + "\\begin{pmatrix}\n", + "1 \\\\\n", + "\\vec{x}_j^{(l-1),\\text{hidden} }\n", + "\\end{pmatrix} \n", + "\\end{aligned}\n", + "$$\n", + "\n", + "The output from the $i$-th neuron at the hidden layer $l$ becomes a vector $\\vec{z}_{i}^{(l),\\ \\text{hidden}}$:\n", + "\n", + "$$\n", + "\\begin{aligned}\n", + "\\vec{z}_{i}^{(l),\\ \\text{hidden}} &= \\Big( b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_1^{(l-1),\\text{hidden} }, \\ \\dots \\ , \\ b_i^{(l), \\ \\text{hidden}} + \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\text{hidden} } \\Big) \\\\\n", + "&= \n", + "\\begin{pmatrix}\n", + "b_i^{(l), \\ \\text{hidden}} & \\big(\\vec{w}_{i}^{(l), \\ \\text{hidden}}\\big)^T\n", + "\\end{pmatrix}\n", + "\\begin{pmatrix}\n", + "1 & 1 & \\dots & 1 \\\\\n", + "\\vec{x}_{1}^{(l-1),\\text{hidden} } & \\vec{x}_{2}^{(l-1),\\text{hidden} } & \\dots & \\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\\text{hidden} }\n", + "\\end{pmatrix}\n", + "\\end{aligned}\n", + "$$\n" + ] + }, + { + "cell_type": "code", + "execution_count": 237, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "def deep_neural_network(deep_params, x):\n", + " # N_hidden is the number of hidden layers \n", + " N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer\n", + " \n", + " # Assumes input x being an one-dimensional array\n", + " num_values = np.size(x)\n", + " x = x.reshape(-1, num_values)\n", + " \n", + " # Assume that the input layer does nothing to the input x\n", + " x_input = x\n", + " \n", + " # Due to multiple hidden layers, define a variable referencing to the\n", + " # output of the previous layer:\n", + " x_prev = x_input \n", + " \n", + " ## Hidden layers:\n", + " \n", + " for l in range(N_hidden):\n", + " # From the list of parameters P; find the correct weigths and bias for this layer\n", + " w_hidden = deep_params[l]\n", + " \n", + " # Add a row of ones to include bias\n", + " x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)\n", + "\n", + " z_hidden = np.matmul(w_hidden, x_prev)\n", + " x_hidden = sigmoid(z_hidden)\n", + "\n", + " # Update x_prev such that next layer can use the output from this layer\n", + " x_prev = x_hidden \n", + "\n", + " ## Output layer:\n", + " \n", + " # Get the weights and bias for this layer\n", + " w_output = deep_params[-1]\n", + " \n", + " # Include bias:\n", + " x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)\n", + "\n", + " z_output = np.matmul(w_output, x_prev)\n", + " x_output = z_output\n", + "\n", + " return x_output" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Backpropagation \n", + "\n", + "This step is very similar for the neural network. The idea in this step is the same as for the neural network, but with more parameters to update for. Again there is no need for computing the gradients analytically since Autograd does the work for us." + ] + }, + { + "cell_type": "code", + "execution_count": 215, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# The trial solution using the deep neural network:\n", + "def g_trial_deep(x,params, g0 = 10):\n", + " return g0 + x*deep_neural_network(params,x)\n", + "\n", + "# The same cost function as for the neural network, but calls deep_neural_network instead.\n", + "def cost_function_deep(P, x):\n", + " \n", + " # Evaluate the trial function with the current parameters P\n", + " g_t = g_trial_deep(x,P)\n", + " \n", + " # Find the derivative w.r.t x of the neural network\n", + " d_net_out = elementwise_grad(deep_neural_network,1)(P,x) \n", + " \n", + " # Find the derivative w.r.t x of the trial function\n", + " d_g_t = elementwise_grad(g_trial_deep,0)(x,P) \n", + " \n", + " # The right side of the ODE \n", + " func = g(x, g_t)\n", + "\n", + " err_sqr = (d_g_t - func)**2\n", + " cost_sum = np.sum(err_sqr)\n", + " \n", + " return cost_sum\n", + "\n", + "def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):\n", + " # num_hidden_neurons is now a list of number of neurons within each hidden layer\n", + "\n", + " # Find the number of hidden layers:\n", + " N_hidden = np.size(num_neurons)\n", + " \n", + " ## Set up initial weigths and biases \n", + " \n", + " # Initialize the list of parameters:\n", + " P = [None]*(N_hidden + 1) # + 1 to include the output layer\n", + "\n", + " P[0] = npr.randn(num_neurons[0], 2 ) \n", + " for l in range(1,N_hidden):\n", + " P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias \n", + " \n", + " # For the output layer\n", + " P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included\n", + "\n", + " print('Initial cost: %g'%cost_function_deep(P, x))\n", + " \n", + " ## Start finding the optimal weigths using gradient descent\n", + " \n", + " # Find the Python function that represents the gradient of the cost function\n", + " # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer\n", + " cost_function_deep_grad = grad(cost_function_deep,0)\n", + " \n", + " # Let the update be done num_iter times\n", + " for i in range(num_iter):\n", + " # Evaluate the gradient at the current weights and biases in P. \n", + " # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases\n", + " # in the hidden layers and output layers evaluated at x.\n", + " cost_deep_grad = cost_function_deep_grad(P, x)\n", + " \n", + " for l in range(N_hidden+1):\n", + " P[l] = P[l] - lmb * cost_deep_grad[l]\n", + "\n", + " print('Final cost: %g'%cost_function_deep(P, x))\n", + " \n", + " return P" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Solving the ODE \n", + "\n", + "Finally, having set up the networks we are ready to use them to solve the ODE problem. \n", + "\n", + "If possible, it is always useful to have an analytical solution at hand to test if the implementations gives reasonable results. \n", + "\n", + "As a recap, the equation to solve is\n", + "\n", + "$$\n", + "g'(x) = -\\gamma g(x) \n", + "$$\n", + "\n", + "where $g(0) = g_0$ with $\\gamma$ and $g_0$ being some chosen values. \n", + "\n", + "Solving this analytically yields\n", + "\n", + "$$\n", + "g(x) = g_0\\exp(-\\gamma x)\n", + "$$\n", + "\n", + "By making the analytical solution availible in our program, it is possible to check the persomance of our neural networks." + ] + }, + { + "cell_type": "code", + "execution_count": 196, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "def g_analytic(x, gamma = 2, g0 = 10):\n", + " return g0*np.exp(-gamma*x)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Using neural network\n", + "\n", + "The code below solves the ODE using a neural network. The number of values for the input $\\vec x$ is 10, number of hidden neurons in the hidden layer being 10 and th step size used in gradien descent $\\lambda = 0.001$. The program updates the weights and biases in the network *num_iter* times. Finally, it plots the results from using the neural network along with the analytical solution. Feel free to experiment with different values and see how the performance of the network is!" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Initial cost: 3670.1\n", + "Final cost: 0.0510011\n" + ] + }, + { + "ename": "NameError", + "evalue": "name 'g_analytic' is not defined", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mg_trial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mP\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0mres_analytical\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mg_analytic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfigsize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mNameError\u001b[0m: name 'g_analytic' is not defined" + ] + } + ], + "source": [ + "npr.seed(15)\n", + "\n", + "## Decide the vales of arguments to the function to solve\n", + "N = 10\n", + "x = np.linspace(0, 1, N)\n", + "\n", + "## Set up the initial parameters\n", + "num_hidden_neurons = 10\n", + "num_iter = 10000\n", + "lmb = 0.001\n", + "\n", + "P = solve_ode_neural_network(x, num_hidden_neurons, num_iter, lmb)\n", + "\n", + "res = g_trial(x,P) \n", + "res_analytical = g_analytic(x)\n", + "\n", + "plt.figure(figsize=(10,10))\n", + "\n", + "plt.title('Performance of neural network solving an ODE compared to the analytical solution')\n", + "plt.plot(x, res_analytical)\n", + "plt.plot(x, res[0,:])\n", + "plt.legend(['analytical','nn'])\n", + "plt.xlabel('x')\n", + "plt.ylabel('g(x)')\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Using a deep neural network\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "npr.seed(15)\n", + "\n", + "## Decide the vales of arguments to the function to solve\n", + "N = 10\n", + "x = np.linspace(0, 1, N)\n", + "\n", + "## Set up the initial parameters\n", + "num_hidden_neurons = np.array([10,10])\n", + "num_iter = 10000\n", + "lmb = 0.001\n", + "\n", + "P = solve_ode_deep_neural_network(x, num_hidden_neurons, num_iter, lmb)\n", + "\n", + "res = g_trial_deep(x,P) \n", + "res_analytical = g_analytic(x)\n", + "\n", + "plt.figure(figsize=(10,10))\n", + "\n", + "plt.title('Performance of a deep neural network solving an ODE compared to the analytical solution')\n", + "plt.plot(x, res_analytical)\n", + "plt.plot(x, res[0,:])\n", + "plt.legend(['analytical','dnn'])\n", + "plt.ylabel('g(x)')\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Wrapping it up \n", + "\n", + "By rewriting the ODE as a minimization problem, it was possible to solve equation using either a neural network (one hidden layer) or a deep neural network (more than one hidden layers). How well the network performed is measured by a specified cost function, which is the function the network tries to minimize. Using a trial solution which satisfies the additional condition and being defined by using the output from the network in some way, the minimization problem could be explicitly defined for out network to solve. The proposed solution from the network is then the trial solution with parameters, that is weights and biases within each layer in the network, such that the solution minimizes the cost function. \n" + ] + } + ], + "metadata": { + "hide_input": false, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.0" + }, + "latex_envs": { + "LaTeX_envs_menu_present": true, + "autoclose": false, + "autocomplete": true, + "bibliofile": "biblio.bib", + "cite_by": "apalike", + "current_citInitial": 1, + "eqLabelWithNumbers": true, + "eqNumInitial": 1, + "hotkeys": { + "equation": "Ctrl-E", + "itemize": "Ctrl-I" + }, + "labels_anchors": false, + "latex_user_defs": false, + "report_style_numbering": false, + "user_envs_cfg": false + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/doc/src/NeuralNet/diffeq.tex b/doc/src/NeuralNet/diffeq.tex new file mode 100644 index 000000000..5897df334 --- /dev/null +++ b/doc/src/NeuralNet/diffeq.tex @@ -0,0 +1,1047 @@ + +% Default to the notebook output style + + + + +% Inherit from the specified cell style. + + + + + +\documentclass[11pt]{article} + + + + \usepackage[T1]{fontenc} + % Nicer default font (+ math font) than Computer Modern for most use cases + \usepackage{mathpazo} + + % Basic figure setup, for now with no caption control since it's done + % automatically by Pandoc (which extracts ![](path) syntax from Markdown). + \usepackage{graphicx} + % We will generate all images so they have a width \maxwidth. This means + % that they will get their normal width if they fit onto the page, but + % are scaled down if they would overflow the margins. + \makeatletter + \def\maxwidth{\ifdim\Gin@nat@width>\linewidth\linewidth + \else\Gin@nat@width\fi} + \makeatother + \let\Oldincludegraphics\includegraphics + % Set max figure width to be 80% of text width, for now hardcoded. + \renewcommand{\includegraphics}[1]{\Oldincludegraphics[width=.8\maxwidth]{#1}} + % Ensure that by default, figures have no caption (until we provide a + % proper Figure object with a Caption API and a way to capture that + % in the conversion process - todo). + \usepackage{caption} + \DeclareCaptionLabelFormat{nolabel}{} + \captionsetup{labelformat=nolabel} + + \usepackage{adjustbox} % Used to constrain images to a maximum size + \usepackage{xcolor} % Allow colors to be defined + \usepackage{enumerate} % Needed for markdown enumerations to work + \usepackage{geometry} % Used to adjust the document margins + \usepackage{amsmath} % Equations + \usepackage{amssymb} % Equations + \usepackage{textcomp} % defines textquotesingle + % Hack from http://tex.stackexchange.com/a/47451/13684: + \AtBeginDocument{% + \def\PYZsq{\textquotesingle}% Upright quotes in Pygmentized code + } + \usepackage{upquote} % Upright quotes for verbatim code + \usepackage{eurosym} % defines \euro + \usepackage[mathletters]{ucs} % Extended unicode (utf-8) support + \usepackage[utf8x]{inputenc} % Allow utf-8 characters in the tex document + \usepackage{fancyvrb} % verbatim replacement that allows latex + \usepackage{grffile} % extends the file name processing of package graphics + % to support a larger range + % The hyperref package gives us a pdf with properly built + % internal navigation ('pdf bookmarks' for the table of contents, + % internal cross-reference links, web links for URLs, etc.) + \usepackage{hyperref} + \usepackage{longtable} % longtable support required by pandoc >1.10 + \usepackage{booktabs} % table support for pandoc > 1.12.2 + \usepackage[inline]{enumitem} % IRkernel/repr support (it uses the enumerate* environment) + \usepackage[normalem]{ulem} % ulem is needed to support strikethroughs (\sout) + % normalem makes italics be italics, not underlines + + + + + % Colors for the hyperref package + \definecolor{urlcolor}{rgb}{0,.145,.698} + \definecolor{linkcolor}{rgb}{.71,0.21,0.01} + \definecolor{citecolor}{rgb}{.12,.54,.11} + + % ANSI colors + \definecolor{ansi-black}{HTML}{3E424D} + \definecolor{ansi-black-intense}{HTML}{282C36} + \definecolor{ansi-red}{HTML}{E75C58} + \definecolor{ansi-red-intense}{HTML}{B22B31} + \definecolor{ansi-green}{HTML}{00A250} + \definecolor{ansi-green-intense}{HTML}{007427} + \definecolor{ansi-yellow}{HTML}{DDB62B} + \definecolor{ansi-yellow-intense}{HTML}{B27D12} + \definecolor{ansi-blue}{HTML}{208FFB} + \definecolor{ansi-blue-intense}{HTML}{0065CA} + \definecolor{ansi-magenta}{HTML}{D160C4} + \definecolor{ansi-magenta-intense}{HTML}{A03196} + \definecolor{ansi-cyan}{HTML}{60C6C8} + \definecolor{ansi-cyan-intense}{HTML}{258F8F} + \definecolor{ansi-white}{HTML}{C5C1B4} + \definecolor{ansi-white-intense}{HTML}{A1A6B2} + + % commands and environments needed by pandoc snippets + % extracted from the output of `pandoc -s` + \providecommand{\tightlist}{% + \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}} + \DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}} + % Add ',fontsize=\small' for more characters per line + \newenvironment{Shaded}{}{} + \newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{\textbf{{#1}}}} + \newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.56,0.13,0.00}{{#1}}} + \newcommand{\DecValTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}} + \newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}} + \newcommand{\FloatTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}} + \newcommand{\CharTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}} + \newcommand{\StringTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}} + \newcommand{\CommentTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textit{{#1}}}} + \newcommand{\OtherTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{{#1}}} + \newcommand{\AlertTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{{#1}}}} + \newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.02,0.16,0.49}{{#1}}} + \newcommand{\RegionMarkerTok}[1]{{#1}} + \newcommand{\ErrorTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{{#1}}}} + \newcommand{\NormalTok}[1]{{#1}} + + % Additional commands for more recent versions of Pandoc + \newcommand{\ConstantTok}[1]{\textcolor[rgb]{0.53,0.00,0.00}{{#1}}} + \newcommand{\SpecialCharTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}} + \newcommand{\VerbatimStringTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}} + \newcommand{\SpecialStringTok}[1]{\textcolor[rgb]{0.73,0.40,0.53}{{#1}}} + \newcommand{\ImportTok}[1]{{#1}} + \newcommand{\DocumentationTok}[1]{\textcolor[rgb]{0.73,0.13,0.13}{\textit{{#1}}}} + \newcommand{\AnnotationTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}} + \newcommand{\CommentVarTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}} + \newcommand{\VariableTok}[1]{\textcolor[rgb]{0.10,0.09,0.49}{{#1}}} + \newcommand{\ControlFlowTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{\textbf{{#1}}}} + \newcommand{\OperatorTok}[1]{\textcolor[rgb]{0.40,0.40,0.40}{{#1}}} + \newcommand{\BuiltInTok}[1]{{#1}} + \newcommand{\ExtensionTok}[1]{{#1}} + \newcommand{\PreprocessorTok}[1]{\textcolor[rgb]{0.74,0.48,0.00}{{#1}}} + \newcommand{\AttributeTok}[1]{\textcolor[rgb]{0.49,0.56,0.16}{{#1}}} + \newcommand{\InformationTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}} + \newcommand{\WarningTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}} + + + % Define a nice break command that doesn't care if a line doesn't already + % exist. + \def\br{\hspace*{\fill} \\* } + % Math Jax compatability definitions + \def\gt{>} + \def\lt{<} + % Document parameters + \title{diffeq} + + + + + % Pygments definitions + +\makeatletter +\def\PY@reset{\let\PY@it=\relax \let\PY@bf=\relax% + \let\PY@ul=\relax \let\PY@tc=\relax% + \let\PY@bc=\relax \let\PY@ff=\relax} +\def\PY@tok#1{\csname PY@tok@#1\endcsname} +\def\PY@toks#1+{\ifx\relax#1\empty\else% + \PY@tok{#1}\expandafter\PY@toks\fi} +\def\PY@do#1{\PY@bc{\PY@tc{\PY@ul{% + \PY@it{\PY@bf{\PY@ff{#1}}}}}}} +\def\PY#1#2{\PY@reset\PY@toks#1+\relax+\PY@do{#2}} + +\expandafter\def\csname PY@tok@w\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.73,0.73}{##1}}} +\expandafter\def\csname PY@tok@c\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}} +\expandafter\def\csname PY@tok@cp\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.74,0.48,0.00}{##1}}} +\expandafter\def\csname PY@tok@k\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@kp\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@kt\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.69,0.00,0.25}{##1}}} +\expandafter\def\csname PY@tok@o\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@ow\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.67,0.13,1.00}{##1}}} +\expandafter\def\csname PY@tok@nb\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@nf\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}} +\expandafter\def\csname PY@tok@nc\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}} +\expandafter\def\csname PY@tok@nn\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}} +\expandafter\def\csname PY@tok@ne\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.82,0.25,0.23}{##1}}} +\expandafter\def\csname PY@tok@nv\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}} +\expandafter\def\csname PY@tok@no\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.53,0.00,0.00}{##1}}} +\expandafter\def\csname PY@tok@nl\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.63,0.63,0.00}{##1}}} +\expandafter\def\csname PY@tok@ni\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.60,0.60,0.60}{##1}}} +\expandafter\def\csname PY@tok@na\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.49,0.56,0.16}{##1}}} +\expandafter\def\csname PY@tok@nt\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@nd\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.67,0.13,1.00}{##1}}} +\expandafter\def\csname PY@tok@s\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@sd\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@si\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.73,0.40,0.53}{##1}}} +\expandafter\def\csname PY@tok@se\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.73,0.40,0.13}{##1}}} +\expandafter\def\csname PY@tok@sr\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.40,0.53}{##1}}} +\expandafter\def\csname PY@tok@ss\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}} +\expandafter\def\csname PY@tok@sx\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@m\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@gh\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,0.50}{##1}}} +\expandafter\def\csname PY@tok@gu\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.50,0.00,0.50}{##1}}} +\expandafter\def\csname PY@tok@gd\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.63,0.00,0.00}{##1}}} +\expandafter\def\csname PY@tok@gi\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.63,0.00}{##1}}} +\expandafter\def\csname PY@tok@gr\endcsname{\def\PY@tc##1{\textcolor[rgb]{1.00,0.00,0.00}{##1}}} +\expandafter\def\csname PY@tok@ge\endcsname{\let\PY@it=\textit} +\expandafter\def\csname PY@tok@gs\endcsname{\let\PY@bf=\textbf} +\expandafter\def\csname PY@tok@gp\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,0.50}{##1}}} +\expandafter\def\csname PY@tok@go\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.53,0.53,0.53}{##1}}} +\expandafter\def\csname PY@tok@gt\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.27,0.87}{##1}}} +\expandafter\def\csname PY@tok@err\endcsname{\def\PY@bc##1{\setlength{\fboxsep}{0pt}\fcolorbox[rgb]{1.00,0.00,0.00}{1,1,1}{\strut ##1}}} +\expandafter\def\csname PY@tok@kc\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@kd\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@kn\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@kr\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@bp\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}} +\expandafter\def\csname PY@tok@fm\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}} +\expandafter\def\csname PY@tok@vc\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}} +\expandafter\def\csname PY@tok@vg\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}} +\expandafter\def\csname PY@tok@vi\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}} +\expandafter\def\csname PY@tok@vm\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}} +\expandafter\def\csname PY@tok@sa\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@sb\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@sc\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@dl\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@s2\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@sh\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@s1\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}} +\expandafter\def\csname PY@tok@mb\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@mf\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@mh\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@mi\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@il\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@mo\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}} +\expandafter\def\csname PY@tok@ch\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}} +\expandafter\def\csname PY@tok@cm\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}} +\expandafter\def\csname PY@tok@cpf\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}} +\expandafter\def\csname PY@tok@c1\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}} +\expandafter\def\csname PY@tok@cs\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}} + +\def\PYZbs{\char`\\} +\def\PYZus{\char`\_} +\def\PYZob{\char`\{} +\def\PYZcb{\char`\}} +\def\PYZca{\char`\^} +\def\PYZam{\char`\&} +\def\PYZlt{\char`\<} +\def\PYZgt{\char`\>} +\def\PYZsh{\char`\#} +\def\PYZpc{\char`\%} +\def\PYZdl{\char`\$} +\def\PYZhy{\char`\-} +\def\PYZsq{\char`\'} +\def\PYZdq{\char`\"} +\def\PYZti{\char`\~} +% for compatibility with earlier versions +\def\PYZat{@} +\def\PYZlb{[} +\def\PYZrb{]} +\makeatother + + + % Exact colors from NB + \definecolor{incolor}{rgb}{0.0, 0.0, 0.5} + \definecolor{outcolor}{rgb}{0.545, 0.0, 0.0} + + + + + % Prevent overflowing lines due to hard-to-break entities + \sloppy + % Setup hyperref package + \hypersetup{ + breaklinks=true, % so long urls are correctly broken across lines + colorlinks=true, + urlcolor=urlcolor, + linkcolor=linkcolor, + citecolor=citecolor, + } + % Slightly bigger margins than the latex defaults + + \geometry{verbose,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in} + + + + \begin{document} + + + \maketitle + + + + + + + \hypertarget{example-exponential-decay-in-one-dimension}{% +\section{Example : Exponential decay in one +dimension}\label{example-exponential-decay-in-one-dimension}} + +In this notebook we will see how a neural network performs when solving +the equation + +\[ +\label{eq:ode} +g'(x) = -\gamma g(x) +\] + +where \(g(0) = g_0\) with \(\gamma\) and \(g_0\) being some chosen +values. This equation is an ordinary differential equation since the +function we have to solve for, \(g(x)\), is of one variable. + +In this example, \(\gamma = 2\) and \(g_0 = 10\) but feel free to change +them and see how the neural network performs. + +\hypertarget{trial-solution}{% +\subsection{Trial solution}\label{trial-solution}} + +To begin with, a trial solution \(g_t(t)\) must be chosen. A general +trial solution for ordinary differential equations could be + +\[ +g_t(x, P) = h_1(x) + h_2(x, N(x, P)) +\] + +with \(h_1(x)\) ensuring that \(g_t(x)\) satisfies some conditions and +\(h_2(x,N(x, P))\) an expression involving \(x\) and the output from the +neural network \(N(x,P)\) with \$P \$ being the collection of the +weights and biases for each layer. It is assumed that there are no +weights and bias at the input layer, so +\(P = \{ P_{\text{hidden}}, P_{\text{output}} \}\). If there are +\(N_{\text{hidden} }\) neurons in the hidden layer, then +\(P_{\text{hidden}}\) is a \(N_{\text{hidden} } \times 2\) matrix. The +first column in \(P_{\text{hidden} }\) represents the bias for each +neuron in the hidden layer and the second column represents the weigths +for each neuron. If there are \(N_{\text{output} }\) neurons in the +output layer, then \$P\_\{\text{output}\} \$ is a +\(N_{\text{output} } \times (1 + N_{\text{hidden} })\) matrix. Its first +column represents the bias of each neuron and the remaining columns +represents the weights to each neuron. + +It is given that \(g(0) = g_0\). The trial solution must fulfill this +condition to be a proper solution of \eqref{eq:ode}. A possible way to +ensure that \(g_t(0, P) = g_0\), is to let \(F(N(x,P)) = x\cdot N(x,P)\) +and \(A(x) = g_0\). This gives the following trial solution: + +\[ +\label{eq:trial} +g_t(x, P) = g_0 + x \cdot N(x, P) +\] + +\hypertarget{reformulating-the-problem}{% +\subsection{Reformulating the problem}\label{reformulating-the-problem}} + +Often, the role of a neural network is to minimize its parameters with +respect to some given error criteria. This criteria, the cost or loss +function, is a measure of how much error the output of the network has +compared to some given known answers. A reformulation of \eqref{eq:ode} +must therefore be done, such that it describes the problem a neural +network can solve. + +The neural network must find the set of weigths and biases \(P\) such +that the trial solution in \eqref{eq:trial} satisfies \eqref{eq:ode}. +The trial solution has been chosen such that it already solves the +condition \(g(0) = g_0\). What remains, is to find \(P\) such that + +\[ +\label{eq:nnmin} +g_t'(x, P) = - \gamma g_t(x, P) +\] + +is fulfilled as \emph{best as possible}. The left hand and right side of +\eqref{eq:nnmin} must be computed seperately, and then the neural +network will choose which weights and biases in \(P\) makes the sides as +equal as possible. Having two sides of an equation as equal as possible, +means that the absolute or squared difference between the sides must be +as close to zero as small. In this case, the difference squared is an +appropiate measurement of how errorneous the trial solution is with +respect to \(P\) of the neural network. Therefore, the problem our +network must solve, is + +\[ +\min_{P}\Big\{ \big(g_t'(x, P) - ( -\gamma g_t(x, P) \big)^2 \Big\} +\] + +or, in terms of weights and biases for each layer: + +\[ +\min_{P_{\text{hidden} }, \ P_{\text{output} }}\Big\{ \big(g_t'(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) - ( -\gamma g_t(x, \{ P_{\text{hidden} }, P_{\text{output} }\}) \big)^2 \Big\} +\] + +for an input value \(x\). If the neural network evaluates \(g_t(x, P)\) +at more avalues for \(x\),~say \(N\) values \(x_i\) for +\(i = 1, \dots, N\), then the \emph{total} error to minimize is + +\[ \label{eq:min} +\min_{P}\Big\{\sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \Big\} +\] + +Letting +\(c(x, P) = \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2\) +denote the cost function, the minimization problem of which our network +must solve, is + +\[ +\min_{P} c(x, P) +\] + +or in terms of \(P_{\text{hidden} }\) and \(P_{\text{output} }\) + +\[ +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\] + +\hypertarget{creating-a-simple-deep-neural-net}{% +\subsection{Creating a simple Deep Neural +Net}\label{creating-a-simple-deep-neural-net}} + +The next step is to decide how the neural net \(N(x, P)\) in +\eqref{eq:trial} should be. In this case, the neural network is made +from scratch to understand better how a neural network works, gain more +control over its architecture, and see how Autograd can be used to +simplify the implementation. + +\hypertarget{an-implementation-of-a-neural-network}{% +\subsubsection{An implementation of a Neural +Network}\label{an-implementation-of-a-neural-network}} + +Since a deep neural network (DNN) is a neural network with more than one +hidden layer, we can first look on how to implement a neural network. +Having an implementation of a neural network at hand, an extension of it +into a deep neural network would (hopefully) be painless. + +For simplicity, it is assumed that the input is an array +\(\vec x = (x_1, \dots, x_N)\) with \(N\) elements. It is at these +points the neural network should find \(P\) such that it fulfills +\eqref{eq:min}. + +\hypertarget{feedforward}{% +\paragraph{Feedforward}\label{feedforward}} + +First, a feedforward of the inputs must be done. This means that +\(\vec x\) must be passed through an input layer, a hidden layer and a +output layer. The input layer in this case, does not need to process the +data any further. The input layer will consist of \(N_{\text{input} }\) +neurons, passing its element to each neuron in the hidden layer. The +number of neurons in the hidden layer will be \(N_{\text{hidden} }\). + +For the \(i\)-th in the hidden layer with weight +\(w_i^{\text{hidden} }\) and bias \(b_i^{\text{hidden} }\), the +weighting from the \(j\)-th neuron at the input layer is: + +\[ +\begin{aligned} +z_{i,j}^{\text{hidden}} &= b_i^{\text{hidden}} + w_i^{\text{hidden}}x_j \\ +&= +\begin{pmatrix} +b_i^{\text{hidden}} & w_i^{\text{hidden}} +\end{pmatrix} +\begin{pmatrix} +1 \\ +x_j +\end{pmatrix} +\end{aligned} +\] + +The result after weighting the input at the \(i\)-th hidden neuron can +be written as a vector: + +\[ +\begin{aligned} +\vec{z}_{i}^{\text{hidden}} &= \Big( b_i^{\text{hidden}} + w_i^{\text{hidden}}x_1 , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_2, \ \dots \, , \ b_i^{\text{hidden}} + w_i^{\text{hidden}} x_N\Big) \\ +&= +\begin{pmatrix} + b_i^{\text{hidden}} & w_i^{\text{hidden}} +\end{pmatrix} +\begin{pmatrix} +1 & 1 & \dots & 1 \\ +x_1 & x_2 & \dots & x_N +\end{pmatrix} \\ +&= \vec{p}_{i, \text{hidden}}^T X +\end{aligned} +\] + +It is the vector \(\vec{p}_{i, \text{hidden}}^T\) that defines each row +in \(P_{\text{hidden} }\), which contains the weights for the neural +network to minimize according to \eqref{eq:min}. + +After having found \$\vec{z}\_\{i\}\^{}\{\text{hidden}\} \$ for every +neuron \(i\) in the hidden layer, the vector will be sent to an +activation function \(a_i(\vec{z})\). In this example, the sigmoid +function has been used: + +\[ +f(z) = \frac{1}{1 + \exp{(-z)}} +\] + +but feel free to chose any activation function you like. + +The output \(\vec{x}_i^{\text{hidden} }\)from each \(i\)-th hidden +neuron is: + +\[ +\vec{x}_i^{\text{hidden} } = f\big( \vec{z}_{i}^{\text{hidden}} \big) +\] + +The outputs \$\vec{x}\_i\^{}\{\text{hidden} \} \$ are then sent to the +output layer. + +The output layer consist of one neuron in this case, and combines the +output from each of the neurons in the hidden layers. The output layer +combines the results from the hidden layer using some weights \$ +w\_i\^{}\{\text{output}\}\$ and biases \(b_i^{\text{output}}\). In this +case, it is assumes that the number of neurons in the output layer is +one. + +The procedure of weigthing the output neuron \(j\) in the hidden layer +to the \(i\)-th neuron in the output layer is similar as for the hidden +layer described previously. + +\[ +\begin{aligned} +z_{1,j}^{\text{output}} & = +\begin{pmatrix} +b_1^{\text{output}} & \vec{w}_1^{\text{output}} +\end{pmatrix} +\begin{pmatrix} +1 \\ +\vec{x}_j^{\text{hidden}} +\end{pmatrix} +\end{aligned} +\] + +Expressing \(z_{1,j}^{\text{output}}\) as a vector gives the following +procedure of weighting the inputs from the hidden layer: + +\[ +\vec{z}_{1}^{\text{output}} = +\begin{pmatrix} +b_1^{\text{output}} & \vec{w}_1^{\text{output}} +\end{pmatrix} +\begin{pmatrix} +1 & 1 & \dots & 1 \\ +\vec{x}_1^{\text{hidden}} & \vec{x}_2^{\text{hidden}} & \dots & \vec{x}_N^{\text{hidden}} +\end{pmatrix} +\] + +In this case we seek a continous range of values since we are +approximating a function. This means that after computing +\(\vec{z}_{1}^{\text{output}}\) the neural network has finished its +feedforward step, and \(\vec{z}_{1}^{\text{output}}\) is the final +output of the network. + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor}3}]:} \PY{c+c1}{\PYZsh{} Autograd will be used for later, so the numpy wrapper for Autograd must be imported} + \PY{k+kn}{import} \PY{n+nn}{autograd}\PY{n+nn}{.}\PY{n+nn}{numpy} \PY{k}{as} \PY{n+nn}{np} + \PY{k+kn}{from} \PY{n+nn}{autograd} \PY{k}{import} \PY{n}{grad}\PY{p}{,} \PY{n}{elementwise\PYZus{}grad} + \PY{k+kn}{import} \PY{n+nn}{autograd}\PY{n+nn}{.}\PY{n+nn}{numpy}\PY{n+nn}{.}\PY{n+nn}{random} \PY{k}{as} \PY{n+nn}{npr} + \PY{k+kn}{from} \PY{n+nn}{matplotlib} \PY{k}{import} \PY{n}{pyplot} \PY{k}{as} \PY{n}{plt} + + \PY{k}{def} \PY{n+nf}{sigmoid}\PY{p}{(}\PY{n}{z}\PY{p}{)}\PY{p}{:} + \PY{k}{return} \PY{l+m+mi}{1}\PY{o}{/}\PY{p}{(}\PY{l+m+mi}{1} \PY{o}{+} \PY{n}{np}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{n}{z}\PY{p}{)}\PY{p}{)} + + \PY{k}{def} \PY{n+nf}{neural\PYZus{}network}\PY{p}{(}\PY{n}{params}\PY{p}{,} \PY{n}{x}\PY{p}{)}\PY{p}{:} + + \PY{c+c1}{\PYZsh{} Find the weights (including and biases) for the hidden and output layer.} + \PY{c+c1}{\PYZsh{} Assume that params is a list of parameters for each layer. } + \PY{c+c1}{\PYZsh{} The biases are the first element for each array in params, } + \PY{c+c1}{\PYZsh{} and the weights are the remaning elements in each array in params. } + + \PY{n}{w\PYZus{}hidden} \PY{o}{=} \PY{n}{params}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]} + \PY{n}{w\PYZus{}output} \PY{o}{=} \PY{n}{params}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]} + + \PY{c+c1}{\PYZsh{} Assumes input x being an one\PYZhy{}dimensional array} + \PY{n}{num\PYZus{}values} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{size}\PY{p}{(}\PY{n}{x}\PY{p}{)} + \PY{n}{x} \PY{o}{=} \PY{n}{x}\PY{o}{.}\PY{n}{reshape}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{num\PYZus{}values}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Assume that the input layer does nothing to the input x} + \PY{n}{x\PYZus{}input} \PY{o}{=} \PY{n}{x} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Hidden layer:} + + \PY{c+c1}{\PYZsh{} Add a row of ones to include bias} + \PY{n}{x\PYZus{}input} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{concatenate}\PY{p}{(}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{ones}\PY{p}{(}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,}\PY{n}{num\PYZus{}values}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{x\PYZus{}input} \PY{p}{)}\PY{p}{,} \PY{n}{axis} \PY{o}{=} \PY{l+m+mi}{0}\PY{p}{)} + + \PY{n}{z\PYZus{}hidden} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{w\PYZus{}hidden}\PY{p}{,} \PY{n}{x\PYZus{}input}\PY{p}{)} + \PY{n}{x\PYZus{}hidden} \PY{o}{=} \PY{n}{sigmoid}\PY{p}{(}\PY{n}{z\PYZus{}hidden}\PY{p}{)} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Output layer:} + + \PY{c+c1}{\PYZsh{} Include bias:} + \PY{n}{x\PYZus{}hidden} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{concatenate}\PY{p}{(}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{ones}\PY{p}{(}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,}\PY{n}{num\PYZus{}values}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{x\PYZus{}hidden} \PY{p}{)}\PY{p}{,} \PY{n}{axis} \PY{o}{=} \PY{l+m+mi}{0}\PY{p}{)} + + \PY{n}{z\PYZus{}output} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{w\PYZus{}output}\PY{p}{,} \PY{n}{x\PYZus{}hidden}\PY{p}{)} + \PY{n}{x\PYZus{}output} \PY{o}{=} \PY{n}{z\PYZus{}output} + + \PY{k}{return} \PY{n}{x\PYZus{}output} +\end{Verbatim} + + + \hypertarget{backpropagation}{% +\paragraph{Backpropagation}\label{backpropagation}} + +Now that feedforward can be done, the next step is to decide how the +parameters should change such that they minimize the cost function. + +Recall that the chosen cost function for this problem is + +\[ +c(x, P) = \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +\] + +In order to minimize it, an optimalization method must be chosen. + +Here, gradient descent with a constant step size has been chosen. + +Before looking at the gradient descent method, let us set up the cost +function along with the right ride of the ODE and trial solution. + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor}4}]:} \PY{c+c1}{\PYZsh{} The trial solution using the deep neural network:} + \PY{k}{def} \PY{n+nf}{g\PYZus{}trial}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{params}\PY{p}{,} \PY{n}{g0} \PY{o}{=} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{:} + \PY{k}{return} \PY{n}{g0} \PY{o}{+} \PY{n}{x}\PY{o}{*}\PY{n}{neural\PYZus{}network}\PY{p}{(}\PY{n}{params}\PY{p}{,}\PY{n}{x}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} The right side of the ODE:} + \PY{k}{def} \PY{n+nf}{g}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{g\PYZus{}trial}\PY{p}{,} \PY{n}{gamma} \PY{o}{=} \PY{l+m+mi}{2}\PY{p}{)}\PY{p}{:} + \PY{k}{return} \PY{o}{\PYZhy{}}\PY{n}{gamma}\PY{o}{*}\PY{n}{g\PYZus{}trial} + + \PY{c+c1}{\PYZsh{} The cost function:} + \PY{k}{def} \PY{n+nf}{cost\PYZus{}function}\PY{p}{(}\PY{n}{P}\PY{p}{,} \PY{n}{x}\PY{p}{)}\PY{p}{:} + + \PY{c+c1}{\PYZsh{} Evaluate the trial function with the current parameters P} + \PY{n}{g\PYZus{}t} \PY{o}{=} \PY{n}{g\PYZus{}trial}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{P}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Find the derivative w.r.t x of the neural network} + \PY{n}{d\PYZus{}net\PYZus{}out} \PY{o}{=} \PY{n}{elementwise\PYZus{}grad}\PY{p}{(}\PY{n}{neural\PYZus{}network}\PY{p}{,}\PY{l+m+mi}{1}\PY{p}{)}\PY{p}{(}\PY{n}{P}\PY{p}{,}\PY{n}{x}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Find the derivative w.r.t x of the trial function} + \PY{n}{d\PYZus{}g\PYZus{}t} \PY{o}{=} \PY{n}{elementwise\PYZus{}grad}\PY{p}{(}\PY{n}{g\PYZus{}trial}\PY{p}{,}\PY{l+m+mi}{0}\PY{p}{)}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{P}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} The right side of the ODE } + \PY{n}{func} \PY{o}{=} \PY{n}{g}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{g\PYZus{}t}\PY{p}{)} + + \PY{n}{err\PYZus{}sqr} \PY{o}{=} \PY{p}{(}\PY{n}{d\PYZus{}g\PYZus{}t} \PY{o}{\PYZhy{}} \PY{n}{func}\PY{p}{)}\PY{o}{*}\PY{o}{*}\PY{l+m+mi}{2} + \PY{n}{cost\PYZus{}sum} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{err\PYZus{}sqr}\PY{p}{)} + + \PY{k}{return} \PY{n}{cost\PYZus{}sum} +\end{Verbatim} + + + \hypertarget{gradient-descent}{% +\subparagraph{Gradient Descent}\label{gradient-descent}} + +The idea of the gradient descent algorithm is to update parameters in +direction where the cost function decreases goes to a minimum. + +In general, the update of some parameters \(\vec \omega\) given a cost +function defined by some weights \(\vec \omega\), \(c(x, \vec \omega)\), +goes as follows: + +\[ +\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega) +\] + +for a number of iterations or until \$ \big\textbar{}\big\textbar{} +\vec \omega\_\{\text{new} \} - +\vec \omega \big\textbar{}\big\textbar{}\$ is smaller than some given +tolerance. + +The value of \(\lambda\) decides how large steps the algorithm must take +in the direction of \$ \nabla\_\{\vec \omega\} c(x, \vec \omega)\$. The +notatation \(\nabla_{\vec \omega}\) denotes the gradient with respect to +the elements in \(\vec \omega\). + +In our case, we have to minimize the cost function \(c(x, P)\) with +respect to the two sets of weights and bisases, that is for the hidden +layer \(P_{\text{hidden} }\) and for the ouput layer +\(P_{\text{output} }\) . + +This means that \(P_{\text{hidden} }\) and \(P_{\text{output} }\) is +updated by + +\[ +\begin{aligned} +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(x, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(x, P) +\end{aligned} +\] + +This might look like a cumberstone to set up the correct expression for +finding the gradients. Luckily, Autograd comes to the rescue. + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor}5}]:} \PY{k}{def} \PY{n+nf}{solve\PYZus{}ode\PYZus{}neural\PYZus{}network}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{num\PYZus{}neurons\PYZus{}hidden}\PY{p}{,} \PY{n}{num\PYZus{}iter}\PY{p}{,} \PY{n}{lmb}\PY{p}{)}\PY{p}{:} + \PY{c+c1}{\PYZsh{}\PYZsh{} Set up initial weigths and biases } + + \PY{c+c1}{\PYZsh{} For the hidden layer} + \PY{n}{p0} \PY{o}{=} \PY{n}{npr}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{n}{num\PYZus{}neurons\PYZus{}hidden}\PY{p}{,} \PY{l+m+mi}{2} \PY{p}{)} + + \PY{c+c1}{\PYZsh{} For the output layer} + \PY{n}{p1} \PY{o}{=} \PY{n}{npr}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{num\PYZus{}neurons\PYZus{}hidden} \PY{o}{+} \PY{l+m+mi}{1} \PY{p}{)} \PY{c+c1}{\PYZsh{} +1 since bias is included} + + \PY{n}{P} \PY{o}{=} \PY{p}{[}\PY{n}{p0}\PY{p}{,} \PY{n}{p1}\PY{p}{]} + + \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Initial cost: }\PY{l+s+si}{\PYZpc{}g}\PY{l+s+s1}{\PYZsq{}}\PY{o}{\PYZpc{}}\PY{k}{cost\PYZus{}function}(P, x)) + + \PY{c+c1}{\PYZsh{}\PYZsh{} Start finding the optimal weigths using gradient descent} + + \PY{c+c1}{\PYZsh{} Find the Python function that represents the gradient of the cost function} + \PY{c+c1}{\PYZsh{} w.r.t the 0\PYZhy{}th input argument \PYZhy{}\PYZhy{} that is the weights and biases in the hidden and output layer} + \PY{n}{cost\PYZus{}function\PYZus{}grad} \PY{o}{=} \PY{n}{grad}\PY{p}{(}\PY{n}{cost\PYZus{}function}\PY{p}{,}\PY{l+m+mi}{0}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Let the update be done num\PYZus{}iter times} + \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n}{num\PYZus{}iter}\PY{p}{)}\PY{p}{:} + \PY{c+c1}{\PYZsh{} Evaluate the gradient at the current weights and biases in P. } + \PY{c+c1}{\PYZsh{} The cost\PYZus{}grad consist now of two arrays; } + \PY{c+c1}{\PYZsh{} one for the gradient w.r.t P\PYZus{}hidden and } + \PY{c+c1}{\PYZsh{} one for the gradient w.r.t P\PYZus{}output} + \PY{n}{cost\PYZus{}grad} \PY{o}{=} \PY{n}{cost\PYZus{}function\PYZus{}grad}\PY{p}{(}\PY{n}{P}\PY{p}{,} \PY{n}{x}\PY{p}{)} + + \PY{n}{P}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]} \PY{o}{=} \PY{n}{P}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]} \PY{o}{\PYZhy{}} \PY{n}{lmb} \PY{o}{*} \PY{n}{cost\PYZus{}grad}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]} + \PY{n}{P}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]} \PY{o}{=} \PY{n}{P}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]} \PY{o}{\PYZhy{}} \PY{n}{lmb} \PY{o}{*} \PY{n}{cost\PYZus{}grad}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]} + + \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Final cost: }\PY{l+s+si}{\PYZpc{}g}\PY{l+s+s1}{\PYZsq{}}\PY{o}{\PYZpc{}}\PY{k}{cost\PYZus{}function}(P, x)) + + \PY{k}{return} \PY{n}{P} +\end{Verbatim} + + + \hypertarget{an-implementation-of-a-deep-neural-network}{% +\subsubsection{An implementation of a Deep Neural +Network}\label{an-implementation-of-a-deep-neural-network}} + +As previously stated, a Deep Neural Network (DNN) follows the same +concept of a neural network, but having more than one hidden layer. +Suppose that the network has \(N_{\text{hidden}}\) hidden layers where +the \(l\)-th layer has \(N_{\text{hidden}}^{(l)}\) neurons. The input is +still assumed to be an array of size \(1 \times N\). The network must +now try to optimalize its output with respect to the collection of +weigths and biases +\(P = \big\{P_{\text{input} }, \ P_{\text{hidden} }^{(1)}, \ P_{\text{hidden} }^{(2)}, \ \dots , \ P_{\text{hidden} }^{(N_{\text{hidden}})}, \ P_{\text{output} }\big\}\). + +\hypertarget{feedforward}{% +\paragraph{Feedforward}\label{feedforward}} + +The feedforward step is similar to as for the neural netowork, but now +considering more than one hidden layer. + +The \(i\)-th neuron at layer \(l\) recieves the result +\(\vec{x}_j^{(l-1),\text{hidden} }\) from the \(j\)-th neuron at layer +\(l-1\). The \(i\)-th neuron at layer \(l\) weights all of the elements +in \(\vec{x}_j^{(l-1),\text{hidden} }\) with a weight vector +\(\vec w_{i,j}^{(l), \ \text{hidden} }\) with as many weigths as there +are elements in\(\vec{x}_j^{(l-1),\text{hidden} }\), and adds a bias +\(b_i^{(l), \ \text{hidden} }\): + +\[ +\begin{aligned} +z_{i,j}^{(l),\ \text{hidden}} &= b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_j^{(l-1),\text{hidden} } \\ +&= +\begin{pmatrix} +b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +\end{pmatrix} +\begin{pmatrix} +1 \\ +\vec{x}_j^{(l-1),\text{hidden} } +\end{pmatrix} +\end{aligned} +\] + +The output from the \(i\)-th neuron at the hidden layer \(l\) becomes a +vector \(\vec{z}_{i}^{(l),\ \text{hidden}}\): + +\[ +\begin{aligned} +\vec{z}_{i}^{(l),\ \text{hidden}} &= \Big( b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_1^{(l-1),\text{hidden} }, \ \dots \ , \ b_i^{(l), \ \text{hidden}} + \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T\vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } \Big) \\ +&= +\begin{pmatrix} +b_i^{(l), \ \text{hidden}} & \big(\vec{w}_{i}^{(l), \ \text{hidden}}\big)^T +\end{pmatrix} +\begin{pmatrix} +1 & 1 & \dots & 1 \\ +\vec{x}_{1}^{(l-1),\text{hidden} } & \vec{x}_{2}^{(l-1),\text{hidden} } & \dots & \vec{x}_{N_{hidden}^{(l-1)}}^{(l-1),\text{hidden} } +\end{pmatrix} +\end{aligned} +\] + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor}237}]:} \PY{k}{def} \PY{n+nf}{deep\PYZus{}neural\PYZus{}network}\PY{p}{(}\PY{n}{deep\PYZus{}params}\PY{p}{,} \PY{n}{x}\PY{p}{)}\PY{p}{:} + \PY{c+c1}{\PYZsh{} N\PYZus{}hidden is the number of hidden layers } + \PY{n}{N\PYZus{}hidden} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{size}\PY{p}{(}\PY{n}{deep\PYZus{}params}\PY{p}{)} \PY{o}{\PYZhy{}} \PY{l+m+mi}{1} \PY{c+c1}{\PYZsh{} \PYZhy{}1 since params consist of parameters to all the hidden layers AND the output layer} + + \PY{c+c1}{\PYZsh{} Assumes input x being an one\PYZhy{}dimensional array} + \PY{n}{num\PYZus{}values} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{size}\PY{p}{(}\PY{n}{x}\PY{p}{)} + \PY{n}{x} \PY{o}{=} \PY{n}{x}\PY{o}{.}\PY{n}{reshape}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{num\PYZus{}values}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Assume that the input layer does nothing to the input x} + \PY{n}{x\PYZus{}input} \PY{o}{=} \PY{n}{x} + + \PY{c+c1}{\PYZsh{} Due to multiple hidden layers, define a variable referencing to the} + \PY{c+c1}{\PYZsh{} output of the previous layer:} + \PY{n}{x\PYZus{}prev} \PY{o}{=} \PY{n}{x\PYZus{}input} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Hidden layers:} + + \PY{k}{for} \PY{n}{l} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n}{N\PYZus{}hidden}\PY{p}{)}\PY{p}{:} + \PY{c+c1}{\PYZsh{} From the list of parameters P; find the correct weigths and bias for this layer} + \PY{n}{w\PYZus{}hidden} \PY{o}{=} \PY{n}{deep\PYZus{}params}\PY{p}{[}\PY{n}{l}\PY{p}{]} + + \PY{c+c1}{\PYZsh{} Add a row of ones to include bias} + \PY{n}{x\PYZus{}prev} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{concatenate}\PY{p}{(}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{ones}\PY{p}{(}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,}\PY{n}{num\PYZus{}values}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{x\PYZus{}prev} \PY{p}{)}\PY{p}{,} \PY{n}{axis} \PY{o}{=} \PY{l+m+mi}{0}\PY{p}{)} + + \PY{n}{z\PYZus{}hidden} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{w\PYZus{}hidden}\PY{p}{,} \PY{n}{x\PYZus{}prev}\PY{p}{)} + \PY{n}{x\PYZus{}hidden} \PY{o}{=} \PY{n}{sigmoid}\PY{p}{(}\PY{n}{z\PYZus{}hidden}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Update x\PYZus{}prev such that next layer can use the output from this layer} + \PY{n}{x\PYZus{}prev} \PY{o}{=} \PY{n}{x\PYZus{}hidden} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Output layer:} + + \PY{c+c1}{\PYZsh{} Get the weights and bias for this layer} + \PY{n}{w\PYZus{}output} \PY{o}{=} \PY{n}{deep\PYZus{}params}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{]} + + \PY{c+c1}{\PYZsh{} Include bias:} + \PY{n}{x\PYZus{}prev} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{concatenate}\PY{p}{(}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{ones}\PY{p}{(}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,}\PY{n}{num\PYZus{}values}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{x\PYZus{}prev}\PY{p}{)}\PY{p}{,} \PY{n}{axis} \PY{o}{=} \PY{l+m+mi}{0}\PY{p}{)} + + \PY{n}{z\PYZus{}output} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{w\PYZus{}output}\PY{p}{,} \PY{n}{x\PYZus{}prev}\PY{p}{)} + \PY{n}{x\PYZus{}output} \PY{o}{=} \PY{n}{z\PYZus{}output} + + \PY{k}{return} \PY{n}{x\PYZus{}output} +\end{Verbatim} + + + \hypertarget{backpropagation}{% +\paragraph{Backpropagation}\label{backpropagation}} + +This step is very similar for the neural network. The idea in this step +is the same as for the neural network, but with more parameters to +update for. Again there is no need for computing the gradients +analytically since Autograd does the work for us. + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor}215}]:} \PY{c+c1}{\PYZsh{} The trial solution using the deep neural network:} + \PY{k}{def} \PY{n+nf}{g\PYZus{}trial\PYZus{}deep}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{params}\PY{p}{,} \PY{n}{g0} \PY{o}{=} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{:} + \PY{k}{return} \PY{n}{g0} \PY{o}{+} \PY{n}{x}\PY{o}{*}\PY{n}{deep\PYZus{}neural\PYZus{}network}\PY{p}{(}\PY{n}{params}\PY{p}{,}\PY{n}{x}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} The same cost function as for the neural network, but calls deep\PYZus{}neural\PYZus{}network instead.} + \PY{k}{def} \PY{n+nf}{cost\PYZus{}function\PYZus{}deep}\PY{p}{(}\PY{n}{P}\PY{p}{,} \PY{n}{x}\PY{p}{)}\PY{p}{:} + + \PY{c+c1}{\PYZsh{} Evaluate the trial function with the current parameters P} + \PY{n}{g\PYZus{}t} \PY{o}{=} \PY{n}{g\PYZus{}trial\PYZus{}deep}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{P}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Find the derivative w.r.t x of the neural network} + \PY{n}{d\PYZus{}net\PYZus{}out} \PY{o}{=} \PY{n}{elementwise\PYZus{}grad}\PY{p}{(}\PY{n}{deep\PYZus{}neural\PYZus{}network}\PY{p}{,}\PY{l+m+mi}{1}\PY{p}{)}\PY{p}{(}\PY{n}{P}\PY{p}{,}\PY{n}{x}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Find the derivative w.r.t x of the trial function} + \PY{n}{d\PYZus{}g\PYZus{}t} \PY{o}{=} \PY{n}{elementwise\PYZus{}grad}\PY{p}{(}\PY{n}{g\PYZus{}trial\PYZus{}deep}\PY{p}{,}\PY{l+m+mi}{0}\PY{p}{)}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{P}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} The right side of the ODE } + \PY{n}{func} \PY{o}{=} \PY{n}{g}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{g\PYZus{}t}\PY{p}{)} + + \PY{n}{err\PYZus{}sqr} \PY{o}{=} \PY{p}{(}\PY{n}{d\PYZus{}g\PYZus{}t} \PY{o}{\PYZhy{}} \PY{n}{func}\PY{p}{)}\PY{o}{*}\PY{o}{*}\PY{l+m+mi}{2} + \PY{n}{cost\PYZus{}sum} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{err\PYZus{}sqr}\PY{p}{)} + + \PY{k}{return} \PY{n}{cost\PYZus{}sum} + + \PY{k}{def} \PY{n+nf}{solve\PYZus{}ode\PYZus{}deep\PYZus{}neural\PYZus{}network}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{num\PYZus{}neurons}\PY{p}{,} \PY{n}{num\PYZus{}iter}\PY{p}{,} \PY{n}{lmb}\PY{p}{)}\PY{p}{:} + \PY{c+c1}{\PYZsh{} num\PYZus{}hidden\PYZus{}neurons is now a list of number of neurons within each hidden layer} + + \PY{c+c1}{\PYZsh{} Find the number of hidden layers:} + \PY{n}{N\PYZus{}hidden} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{size}\PY{p}{(}\PY{n}{num\PYZus{}neurons}\PY{p}{)} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Set up initial weigths and biases } + + \PY{c+c1}{\PYZsh{} Initialize the list of parameters:} + \PY{n}{P} \PY{o}{=} \PY{p}{[}\PY{k+kc}{None}\PY{p}{]}\PY{o}{*}\PY{p}{(}\PY{n}{N\PYZus{}hidden} \PY{o}{+} \PY{l+m+mi}{1}\PY{p}{)} \PY{c+c1}{\PYZsh{} + 1 to include the output layer} + + \PY{n}{P}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]} \PY{o}{=} \PY{n}{npr}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{n}{num\PYZus{}neurons}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]}\PY{p}{,} \PY{l+m+mi}{2} \PY{p}{)} + \PY{k}{for} \PY{n}{l} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,}\PY{n}{N\PYZus{}hidden}\PY{p}{)}\PY{p}{:} + \PY{n}{P}\PY{p}{[}\PY{n}{l}\PY{p}{]} \PY{o}{=} \PY{n}{npr}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{n}{num\PYZus{}neurons}\PY{p}{[}\PY{n}{l}\PY{p}{]}\PY{p}{,} \PY{n}{num\PYZus{}neurons}\PY{p}{[}\PY{n}{l}\PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{]} \PY{o}{+} \PY{l+m+mi}{1}\PY{p}{)} \PY{c+c1}{\PYZsh{} +1 to include bias } + + \PY{c+c1}{\PYZsh{} For the output layer} + \PY{n}{P}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{]} \PY{o}{=} \PY{n}{npr}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{num\PYZus{}neurons}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{]} \PY{o}{+} \PY{l+m+mi}{1} \PY{p}{)} \PY{c+c1}{\PYZsh{} +1 since bias is included} + + \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Initial cost: }\PY{l+s+si}{\PYZpc{}g}\PY{l+s+s1}{\PYZsq{}}\PY{o}{\PYZpc{}}\PY{k}{cost\PYZus{}function\PYZus{}deep}(P, x)) + + \PY{c+c1}{\PYZsh{}\PYZsh{} Start finding the optimal weigths using gradient descent} + + \PY{c+c1}{\PYZsh{} Find the Python function that represents the gradient of the cost function} + \PY{c+c1}{\PYZsh{} w.r.t the 0\PYZhy{}th input argument \PYZhy{}\PYZhy{} that is the weights and biases in the hidden and output layer} + \PY{n}{cost\PYZus{}function\PYZus{}deep\PYZus{}grad} \PY{o}{=} \PY{n}{grad}\PY{p}{(}\PY{n}{cost\PYZus{}function\PYZus{}deep}\PY{p}{,}\PY{l+m+mi}{0}\PY{p}{)} + + \PY{c+c1}{\PYZsh{} Let the update be done num\PYZus{}iter times} + \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n}{num\PYZus{}iter}\PY{p}{)}\PY{p}{:} + \PY{c+c1}{\PYZsh{} Evaluate the gradient at the current weights and biases in P. } + \PY{c+c1}{\PYZsh{} The cost\PYZus{}grad consist now of N\PYZus{}hidden + 1 arrays; the gradient w.r.t the weights and biases} + \PY{c+c1}{\PYZsh{} in the hidden layers and output layers evaluated at x.} + \PY{n}{cost\PYZus{}deep\PYZus{}grad} \PY{o}{=} \PY{n}{cost\PYZus{}function\PYZus{}deep\PYZus{}grad}\PY{p}{(}\PY{n}{P}\PY{p}{,} \PY{n}{x}\PY{p}{)} + + \PY{k}{for} \PY{n}{l} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n}{N\PYZus{}hidden}\PY{o}{+}\PY{l+m+mi}{1}\PY{p}{)}\PY{p}{:} + \PY{n}{P}\PY{p}{[}\PY{n}{l}\PY{p}{]} \PY{o}{=} \PY{n}{P}\PY{p}{[}\PY{n}{l}\PY{p}{]} \PY{o}{\PYZhy{}} \PY{n}{lmb} \PY{o}{*} \PY{n}{cost\PYZus{}deep\PYZus{}grad}\PY{p}{[}\PY{n}{l}\PY{p}{]} + + \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Final cost: }\PY{l+s+si}{\PYZpc{}g}\PY{l+s+s1}{\PYZsq{}}\PY{o}{\PYZpc{}}\PY{k}{cost\PYZus{}function\PYZus{}deep}(P, x)) + + \PY{k}{return} \PY{n}{P} +\end{Verbatim} + + + \hypertarget{solving-the-ode}{% +\subsection{Solving the ODE}\label{solving-the-ode}} + +Finally, having set up the networks we are ready to use them to solve +the ODE problem. + +If possible, it is always useful to have an analytical solution at hand +to test if the implementations gives reasonable results. + +As a recap, the equation to solve is + +\[ +g'(x) = -\gamma g(x) +\] + +where \(g(0) = g_0\) with \(\gamma\) and \(g_0\) being some chosen +values. + +Solving this analytically yields + +\[ +g(x) = g_0\exp(-\gamma x) +\] + +By making the analytical solution availible in our program, it is +possible to check the persomance of our neural networks. + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor}196}]:} \PY{k}{def} \PY{n+nf}{g\PYZus{}analytic}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{gamma} \PY{o}{=} \PY{l+m+mi}{2}\PY{p}{,} \PY{n}{g0} \PY{o}{=} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{:} + \PY{k}{return} \PY{n}{g0}\PY{o}{*}\PY{n}{np}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{n}{gamma}\PY{o}{*}\PY{n}{x}\PY{p}{)} +\end{Verbatim} + + + \hypertarget{using-neural-network}{% +\subsubsection{Using neural network}\label{using-neural-network}} + +The code below solves the ODE using a neural network. The number of +values for the input \(\vec x\) is 10, number of hidden neurons in the +hidden layer being 10 and th step size used in gradien descent +\(\lambda = 0.001\). The program updates the weights and biases in the +network \emph{num\_iter} times. Finally, it plots the results from using +the neural network along with the analytical solution. Feel free to +experiment with different values and see how the performance of the +network is! + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor}6}]:} \PY{n}{npr}\PY{o}{.}\PY{n}{seed}\PY{p}{(}\PY{l+m+mi}{15}\PY{p}{)} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Decide the vales of arguments to the function to solve} + \PY{n}{N} \PY{o}{=} \PY{l+m+mi}{10} + \PY{n}{x} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{linspace}\PY{p}{(}\PY{l+m+mi}{0}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{,} \PY{n}{N}\PY{p}{)} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Set up the initial parameters} + \PY{n}{num\PYZus{}hidden\PYZus{}neurons} \PY{o}{=} \PY{l+m+mi}{10} + \PY{n}{num\PYZus{}iter} \PY{o}{=} \PY{l+m+mi}{10000} + \PY{n}{lmb} \PY{o}{=} \PY{l+m+mf}{0.001} + + \PY{n}{P} \PY{o}{=} \PY{n}{solve\PYZus{}ode\PYZus{}neural\PYZus{}network}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{num\PYZus{}hidden\PYZus{}neurons}\PY{p}{,} \PY{n}{num\PYZus{}iter}\PY{p}{,} \PY{n}{lmb}\PY{p}{)} + + \PY{n}{res} \PY{o}{=} \PY{n}{g\PYZus{}trial}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{P}\PY{p}{)} + \PY{n}{res\PYZus{}analytical} \PY{o}{=} \PY{n}{g\PYZus{}analytic}\PY{p}{(}\PY{n}{x}\PY{p}{)} + + \PY{n}{plt}\PY{o}{.}\PY{n}{figure}\PY{p}{(}\PY{n}{figsize}\PY{o}{=}\PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,}\PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)} + + \PY{n}{plt}\PY{o}{.}\PY{n}{title}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Performance of neural network solving an ODE compared to the analytical solution}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{res\PYZus{}analytical}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{res}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{,}\PY{p}{:}\PY{p}{]}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{legend}\PY{p}{(}\PY{p}{[}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{analytical}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{nn}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{xlabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{x}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{ylabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{g(x)}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)} +\end{Verbatim} + + + \begin{Verbatim}[commandchars=\\\{\}] +Initial cost: 3670.1 +Final cost: 0.0510011 + + \end{Verbatim} + + \begin{Verbatim}[commandchars=\\\{\}] + + --------------------------------------------------------------------------- + + NameError Traceback (most recent call last) + + in () + 13 + 14 res = g\_trial(x,P) + ---> 15 res\_analytical = g\_analytic(x) + 16 + 17 plt.figure(figsize=(10,10)) + + + NameError: name 'g\_analytic' is not defined + + \end{Verbatim} + + \hypertarget{using-a-deep-neural-network}{% +\subsubsection{Using a deep neural +network}\label{using-a-deep-neural-network}} + + \begin{Verbatim}[commandchars=\\\{\}] +{\color{incolor}In [{\color{incolor} }]:} \PY{n}{npr}\PY{o}{.}\PY{n}{seed}\PY{p}{(}\PY{l+m+mi}{15}\PY{p}{)} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Decide the vales of arguments to the function to solve} + \PY{n}{N} \PY{o}{=} \PY{l+m+mi}{10} + \PY{n}{x} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{linspace}\PY{p}{(}\PY{l+m+mi}{0}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{,} \PY{n}{N}\PY{p}{)} + + \PY{c+c1}{\PYZsh{}\PYZsh{} Set up the initial parameters} + \PY{n}{num\PYZus{}hidden\PYZus{}neurons} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{array}\PY{p}{(}\PY{p}{[}\PY{l+m+mi}{10}\PY{p}{,}\PY{l+m+mi}{10}\PY{p}{]}\PY{p}{)} + \PY{n}{num\PYZus{}iter} \PY{o}{=} \PY{l+m+mi}{10000} + \PY{n}{lmb} \PY{o}{=} \PY{l+m+mf}{0.001} + + \PY{n}{P} \PY{o}{=} \PY{n}{solve\PYZus{}ode\PYZus{}deep\PYZus{}neural\PYZus{}network}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{num\PYZus{}hidden\PYZus{}neurons}\PY{p}{,} \PY{n}{num\PYZus{}iter}\PY{p}{,} \PY{n}{lmb}\PY{p}{)} + + \PY{n}{res} \PY{o}{=} \PY{n}{g\PYZus{}trial\PYZus{}deep}\PY{p}{(}\PY{n}{x}\PY{p}{,}\PY{n}{P}\PY{p}{)} + \PY{n}{res\PYZus{}analytical} \PY{o}{=} \PY{n}{g\PYZus{}analytic}\PY{p}{(}\PY{n}{x}\PY{p}{)} + + \PY{n}{plt}\PY{o}{.}\PY{n}{figure}\PY{p}{(}\PY{n}{figsize}\PY{o}{=}\PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,}\PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)} + + \PY{n}{plt}\PY{o}{.}\PY{n}{title}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Performance of a deep neural network solving an ODE compared to the analytical solution}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{res\PYZus{}analytical}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{x}\PY{p}{,} \PY{n}{res}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{,}\PY{p}{:}\PY{p}{]}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{legend}\PY{p}{(}\PY{p}{[}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{analytical}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{dnn}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{ylabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{g(x)}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)} + \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)} +\end{Verbatim} + + + \hypertarget{wrapping-it-up}{% +\subsection{Wrapping it up}\label{wrapping-it-up}} + +By rewriting the ODE as a minimization problem, it was possible to solve +equation using either a neural network (one hidden layer) or a deep +neural network (more than one hidden layers). How well the network +performed is measured by a specified cost function, which is the +function the network tries to minimize. Using a trial solution which +satisfies the additional condition and being defined by using the output +from the network in some way, the minimization problem could be +explicitly defined for out network to solve. The proposed solution from +the network is then the trial solution with parameters, that is weights +and biases within each layer in the network, such that the solution +minimizes the cost function. + + + % Add a bibliography block to the postdoc + + + + \end{document}