GUANinE v1.0: Benchmark Datasets for Genomic AI Sequence-to-Function Models

Abstract

Computational genomics increasingly relies on machine learning methods for genome interpretation, and the recent adoption of neural sequence-to-function models highlights the need for rigorous model specification and controlled evaluation, problems familiar to other fields of AI. Research strategies that have greatly benefited other fields — including benchmarking, auditing, and algorithmic fairness — are also needed to advance the field of genomic AI and to facilitate model development. Here we propose a genomic AI benchmark,GUANinE, for evaluating model generalization across a number of distinct genomic tasks. Compared to existing task formulations in computational genomics,GUANinEis large-scale, de-noised, and suitable for evaluating pretrained models.GUANinEv0.9 primarily focuses on functional genomics tasks such as functional element annotation and gene expression prediction, but also draws upon connections to evolutionary biology through sequence conservation tasks. The currentGUANinEtasks provide insight into the performance of existing genomic AI models and non-neural baselines, with opportunities to be refined, revisited, and broadened as the field matures. Finally, theGUANinEbenchmark allows us to evaluate new self-supervised T5 models and explore the tradeoffs between tokenization and model performance, while showcasing the potential for self-supervision to complement existing pretraining procedures.