Accuracy of a neural network differential equation solver

Abstract

Modeling complex systems is basically a two-fold process. The first step involves formulation of the dynamical relations which usually takes the form of differential equations (DE), while the second step engages in determination of the relationship between dynamical variables. Each of the tasks is significant and oftentimes, equally difficult. For example, when the relevant DEs of interacting systems are prepared, say the Lotka-Volterra equations for predator-prey models or the hydrodynamical equations for fluid motion, the problem of solving this equation is equally formidable since in many cases the DE has no known analytic solution. One way of solving a DE is by transforming its coordinates such that it resembles a form with known analytic solution. Such a process is however generally tedious, or at worst, will never work because such a transformation does not exist in the first place. A more common approach is by utilizing numerical techniques (e.g. finite-difference, Runge Kutta, etc.) which generally relies on the accuracy of the sampling interval. However, numerical methods provide an iterative representation of the solution and hence, the error propagates at later iterations. Another way is by using neural networks (NN). Recently we have shown that an unsupervised NN trained using the modified backpropagation method can solve a differential equation that: 1) models the propagation of pulses in nonlinear media, 2) replicates the arrangement of competing biological entities in a given space, and 3) determines the optimum design for a nuclear reactor. In this study we analyze formally how accurately an NN can solve linear DEs and we propose an approach in increasing the accuracy of the solution obtained by NN for general types of DEs. We utilize a three layer unsupervised NN with inputs {x₁, x₂, ... , x_n} in solving the DE which is given by F(x₁, x₂, ... , x_n) = DΨ (x₁, x₂, ... , x_n) = 0, where D is a differential operator and the x_i are the dynamical parameters. The s^th output Ψ_s^(k), corresponding to the k^th set of inputs of the NN, is given by Ψ_s^(k) =f₀(∑_m=1,H d_sm^(k)y_m^(k)); y_m^(k) = f₀(∑_j=1,L w_mj^(k)r_j) where r_j is the j^th input, y_m^(k) gives the m^th output of the hidden nodes {m = 1, 2, ... , H}, f_H(z) = tanh(z) and f₀(z) = z represents the hidden and output activation functions, respectively. Parameter w_mj^(k) describes the interconnection weights between neurons in the m^th and the j^th layer while d_sm^(k) gives the synaptic strength of the s^th and m^th layer.