Enhanced throughput message streaming methods for DAQ systems of physics experiments

Abstract

In nuclear and particle experiments, to achieve a real-time software-based data acquisition (DAQ) system, a message streaming platform is adopted as a solution for real-time data streaming transmission, facilitating a decoupled and scalable architecture. As increasing data rate in recent experiments demands higher and higher throughput, existing message streaming platforms lack of high single-node throughput, guarantees of reliability, and sequentiality for high-throughput transmission applications. This paper introduces enhanced throughput message streaming methods for DAQ systems, named DAQMQ. DAQMQ utilizes in-memory storage to store messages during transmission, offering high throughput performance and configurable message retention capability. Reliability in high-throughput transmission is strictly guaranteed by employing a two-phased acknowledgment mechanism. By fast reorganizing message streams in real-time, DAQMQ achieves high-throughput non-blocking sequential transmission. Evaluations conducted on ordinary PCs with 10-Gigabit network connections show that DAQMQ achieves a stable single-node throughput of 9.0 Gbit/s even when handling up to 1024 topics. Furthermore, DAQMQ achieves sequential transmission throughput close to that of non-sequential transmission. Tests with a broker or consumer breakdown during high-throughput transmission demonstrate DAQMQ's reliability and asynchronous transmission ability.