DisruptionBench: A robust benchmarking framework for machine learning-driven disruption prediction

Abstract

Disruptions remain a major challenge to commercialization of fusion leveraging the tokamak configuration. Machine Learning (ML) holds promise for prediction in its flexibility and knowledge transfer between devices. However, the current body of ML-driven disruption prediction research lacks systematic benchmarking standards. This work introduces a novel benchmarking platform -- DisruptionBench -- encompassing nine modeling tasks tailored onto real-time tokamak operations: the cross of three test tokamaks (Alcator C-Mod, DIII-D, and EAST) with three test regimes (zero-shot, few-shot, and many-shot.) We conduct a comprehensive evaluation of four ML models re-implementing published ones as well as introducing two novel approaches – a GPT-2-like transformer and the Continuous Convolutional Neural Network (CCNN). Across all evaluation metrics, the CCNN demonstrates the most consistent and superior performance (best AUC=0.974), using the Alcator C-Mod tokamak as a test dataset. However, the GPT-2 architecture (best AUC=0.84) and the Hybrid Deep Learner (best AUC=0.92) re-implementation succeed in specific cases. Exploring the role of ''memory" in the transformer model, we observe that it often learns to recognize shot initiation, suggesting the potential utility of models with broader temporal context. Finally, as modeling demands for nuclear fusion datasets enter an era of unprecedented size and complexity, we characterize a newly compiled dataset spanning the entire operational history of Alcator C-Mod (1991-2016). These benchmarking, modeling, and data resources promise to accelerate the development of advanced ML-driven disruption prediction capabilities.