A Simple and Efficient Parallel Approach to Large-Scale Railway Freight Data Analysis

Abstract

Analysis of rail freight data is essential to make a better strategy for railway freight traffic organization. However, rail freight data are national-widely stored in data center of railway bureaus. Furthermore, the size of the data is very large and grows rapidly. Although analytical data processing has received much attention these years, little attention is paid to transparently processing data stored in remote data sources, particularly where the data are generated. In this paper, we present a simple and efficient parallel approach, RFDPS, to large-scale rail freight data processing. RFDPS provides a novel virtualization data model to integrate geographically distributed freight data. Besides, it can parallelize and optimize data processing automatically across multiple data center of railway bureaus. We run typical rail freight analysis applications in production environment to evaluate the efficiency of RFDPS. The results show that the performance of RFDPS is improved exponentially compared to the approach China's Ministry of Railways adopted before.