A high‐performance distributed file system for large‐scale concurrent HD video streams

Abstract

SUMMARYWith the rapid development of intelligent transportation technology, high-definition data storage, and processing of massive amounts of video surveillance have become key issues. When thousands of high-bit-rate video streaming concurrently writes, disk I/O throughput becomes a bottleneck. In addition, this leads to serious energy consumption and disk abrasion. To solve these problems, a new distributed file system for high concurrent and high-bit-rate writing is designed. It combines an optimized data storage model, efficient metadata management, and exquisite disk schedule mechanism. The optimized data storage model uses a file pre-allocation strategy and multiple-stream input modulating technology to convert the randomly concurrent writes on the disk into sequential writes; the metadata management provides an efficient means of retrieving the specified data; and the dual-partition schedule mechanism can ensure the disk stability with less abrasion. Through this distributed file system, the disk I/O throughput can be saturated in a high concurrent writing environment. The performance evaluation results demonstrate that the I/O throughput of a normal 7200 RPM SATA III disk in our scheme can be stabilized at 150 MB/s, easily to support 300 concurrent high-definition video streams (4 Mbit/s each). The distributed file system with eight commodity servers can afford the ability of supporting 8000 high-definition video streams concurrently writing, which is far greater than the existing video surveillance storage solutions.

Magpie: Automatically Tuning Static Parameters for Distributed File Systems using Deep Reinforcement Learning

“…In this experiment, we evaluate the tuning performance under five FileBench workloads. File Server and Video Server are commonly used with DFS [32], and the other three workloads are regular IO operations in DFS. As there is no existing automatic parameter tuning system that can directly tune static parameters, we employ a general parameter tuning system, BestConfig [27], as our baseline.…”

Advances in Parallel and Distributed Computing and Communications

“…The paper A High-Performance Distributed File System For Large-Scale Concurrent HD Video Streams, by Duan et al [8], designs a new distributed file system for high concurrent and high-bitrate writing. The performance evaluation results demonstrate that the I/O throughput of a normal 7200 RPM SATA III disc in the proposed scheme can be stabilised at 150 MB/s.…”