Gene regulatory network (GRN) could provide guidance for understanding the internal laws of biological phenomena and analyzing several diseases. Ordinary differential equation model, which owns continuity and flexibility, has been utilized to identify GRN over the past decade. In this paper, we propose a novel algorithm, which is named as RNDEtree, a nonlinear ordinary differential equation model based on a flexible neural tree to improve the accuracy of the GRN reconstruction. In this model, a flexible neural tree can be utilized to approximate the nonlinear regulation function of an ordinary differential equation model. Multiexpression programming is proposed to evolve the structure of a flexible neural tree, and the brainstorm optimization algorithm is utilized to optimize the parameters of the RNDEtree model. In order to improve the false-positive ratio of this method, a novel fitness function is proposed, in which sparse and minimum redundancy maximum relevance (mRMR) terms are considered when optimizing RNDEtree. The performances of our proposed algorithm can be evaluated by the benchmark datasets from the DREAM challenge and real biological dataset in E. coli. The experimental results demonstrate that the proposed method could infer more correctly GRN than the other state-the-art methods.INDEX TERMS Gene regulatory network, flexible neural tree model, ordinary differential equation, mutual information, minimum redundancy maximum relevance.