Distributed Transcoding System Integrating HDFS and MapReduce

Abstract

With the development of TV data services, the video conversion equipment in the past was concentrated, and the storage capacity and processing functions did not have scalability, and could not adapt to large-scale video processing. The video editing task is an information-intensive task, and a large number of video transcoding tasks are transferred to a distributed system, making large-scale video transcoding work possible. The purpose of this article is to integrate the research of HDFS and MapReduce distributed transcoding system, fully consider the advantages of HDFS and MapReduce, combine video encoding and decoding technology and video submission method, and propose a group-based classification GOP. Video clips are transcoded individually, and the entire transcoding system is easily configured and managed through a workflow. The results show the impact of changing the number of cluster nodes on transcoding performance, with 10 and 50 GB datasets having faster transcoding speeds in the system compared to 1, 2, 4 and 6 GB datasets. Effects of changing block size and manual copy number on transcoding performance. The experimental results show that the system achieves the best transcoding time performance when the block copy number is set to 4.