Functional Models of Hadoop MapReduce with Application to Scan

Abstract

MapReduce, first proposed by Google, is a remarkable programming model for processing very large amounts of data. An open-source implementation of MapReduce, called Hadoop, is now used for developing a wide range of applications. Although developing a correct and efficient program on MapReduce is much easier than developing one with MPI etc., it is still nontrivial if the target application requires involved functionalities of Hadoop MapReduce. Under these situations, functional models for MapReduce computation play important roles because we can utilize them for better understanding, proving the correctness, and even optimization of MapReduce programs. In this paper, we develop two functional models, a lowlevel one and a high-level one, which capture the semantics of Hadoop MapReduce computation. We discuss the detailed semantics mainly in terms of the following two computations: the computation of Mapper and Reducer classes and the computation in the Shuffle phase with the secondary-sorting technique. In addition, we develop MapReduce algorithms for the scan computational pattern (prefix sums) on the newly proposed models.

An implementation of real-time air quality and influenza-like illness data storage and processing platform

“…HDFS is a decentralized storage system which provides reliable, scalable and faults tolerant data storage (Jena, Gourisaria, Rautaray, & Pandey, 2017). It works with MapReduce computing framework based on the requests and queries (Matsuzaki, 2017;Park, Min, & Shim, 2017;Pulgar-Rubio, Rivera-Rivas, Prez-Godoy, Gonzlez, Carmona, & del Jesus, 2017;Sha, Liang, Jiang, Chen, & Zhuge, 2017, pp. 585-592).…”