Research on the communication strategy of data hub

Abstract

The Open Data Cube (ODC) is an open-source platform for managing, analyzing, and visualizing Earth observation data, especially satellite Remote Sensing data. We proposed the Data Hub system because a single Data Cube cannot undertake the analysis and research works across various remote sensing domains. The Data Hub is an integrated system built upon multiple Data Cubes. This system is specifically designed to manage multiple Data Cubes and perform tasks such as data management, indexing, and processing of remote sensing data. This paper mainly focuses on the strategy of Data Hub communication with multiple Data Cubes. Communication includes two parts: KeepALive (heartbeat) transfer and metadata synchronization. In order to minimize the network workload on the Data Hub system and identify the optimal communication strategy, we conducted two experiments: experiments on KeepALive frequency and experiments on metadata synchronization strategies. We designed comparative experiments to simulate the computational and metadata synchronization tasks between the Data Hub server and multiple Data Cube servers. Based on the experimental results, we identified the optimal frequency for sending KeepALive signals and gained insights into the impact of network conditions on metadata synchronization strategies (Pull strategy, Push strategy, and Pull-Push hybrid strategy). Finally, we determined the sending frequency of KeepALive packets to be once every three hours and selected the pull-push hybrid strategy as the transfer strategy for the metadata synchronization module.