Symbolic Regression via Neural-Guided Genetic Programming Population Seeding

Abstract

Symbolic regression is the process of identifying mathematical expressions that fit observed output from a black-box process. It is a discrete optimization problem generally believed to be NP-hard. Prior approaches to solving the problem include neural-guided search (e.g. using reinforcement learning) and genetic programming. In this work, we introduce a hybrid neural-guided/genetic programming approach to symbolic regression and other combinatorial optimization problems. We propose a neural-guided component used to seed the starting population of a random restart genetic programming component, gradually learning better starting populations. On a number of common benchmark tasks to recover underlying expressions from a dataset, our method recovers 65% more expressions than a recently published top-performing model using the same experimental setup. We demonstrate that running many genetic programming generations without interdependence on the neural-guided component performs better for symbolic regression than alternative formulations where the two are more strongly coupled. Finally, we introduce a new set of 22 symbolic regression benchmark problems with increased difficulty over existing benchmarks. Source code is provided at www.github.com/brendenpetersen/deep-symbolic-optimization.

Efficient Generator of Mathematical Expressions for Symbolic Regression

“…EDHiE can then evaluate the fit of the obtained expressions against the measurements. We conjecture that the performance of EDHiE on standard benchmarks [13,14] would compare favorably to that of a state-ofthe-art symbolic regression method [15]. The results of our empirical evaluation of HVAE and EDHiE confirm our conjectures.…”

“…Recently, many symbolic regression approaches based on neural networks have been proposed [14,[25][26][27][28][29]. In particular, Deep Symbolic Optimization, DSO combines autoencoders with genetic programming to approach symbolic regression, among other combinatorial optimization tasks [15]. The neural networks are used to sample the individuals in the initial population of the evolutionary algorithm and are retrained at each iteration to focus on expressions leading to better fit.…”

End-to-end symbolic regression with transformers

“…Figure 12: Illustration of our model on a few benchmark datasets from the litterature. We show the prediction of our model on six 2-dimensional datasets presented in [36] and used as a comparison point in a few recent works [37]. The input points are marked as black crosses.…”

CoNSoLe: Convex Neural Symbolic Learning

“…Symbolic regression using neural networks. As discussed in Section 1, the high approximation capacity of NNs can facilitate SR in both the symbol search [3,[12][13][14][15] and the expression representation [6][7][8][9][10]. In addition, there are studies to indirectly help SR with NNs.…”

“…The second group employs NNs to search the symbol connections, and non-linear optimizations like BFGS [11] can be employed to estimate symbol coefficients. For the searching procedure, [3,[12][13][14] leverage a Recursive Neural Network (RNN) as a policy network to iteratively generate optimal actions that can select and connect symbols. [15] employs large-scale pre-training to directly map from data to the symbolic equations.…”