Efficiently support MapReduce-like computation models inside parallel DBMS

Chen, Qiming; Therber, Andy; Hsu, Meichun; Zeller, Hans; Zhang, Bin; Ren, Wu

doi:10.1145/1620432.1620438

Cited by 9 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, our focus is on MapReduce programs written directly in Java. The database community has also been interested in several other aspects of bridging the gap between MapReduce and DBMSs [15].…”

Section: Related Workmentioning

confidence: 99%

Column-oriented storage techniques for MapReduce

et al. 2011

View full text Add to dashboard Cite

Users of MapReduce often run into performance problems when they scale up their workloads. Many of the problems they encounter can be overcome by applying techniques learned from over three decades of research on parallel DBMSs. However, translating these techniques to a MapReduce implementation such as Hadoop presents unique challenges that can lead to new design choices. This paper describes how column-oriented storage techniques can be incorporated in Hadoop in a way that preserves its popular programming APIs.We show that simply using binary storage formats in Hadoop can provide a 3x performance boost over the naive use of text files. We then introduce a column-oriented storage format that is compatible with the replication and scheduling constraints of Hadoop and show that it can speed up MapReduce jobs on real workloads by an order of magnitude. We also show that dealing with complex column types such as arrays, maps, and nested records, which are common in MapReduce jobs, can incur significant CPU overhead. Finally, we introduce a novel skip list column format and lazy record construction strategy that avoids deserializing unwanted records to provide an additional 1.5x performance boost. Experiments on a real intranet crawl are used to show that our column-oriented storage techniques can improve the performance of the map phase in Hadoop by as much as two orders of magnitude.

show abstract

Section: Related Workmentioning

confidence: 99%

Column-oriented storage techniques for MapReduce

et al. 2011

View full text Add to dashboard Cite

show abstract

“…In the Web age with the massively growing data volume and the pressing need for low latency, one option for the next generation BI system to scale up data intensive analytics is to push computation down to the data management layer for fast data access and reduced data transfer [6,10,13,14,22]. The major effort for carrying out analytics inside the DB engine is wrapping computations by User Defined Functions (UDFs) executed in the query processing environment [1,10,12,15,19].…”

Section: Introductionmentioning

confidence: 99%

“…The major effort for carrying out analytics inside the DB engine is wrapping computations by User Defined Functions (UDFs) executed in the query processing environment [1,10,12,15,19].…”

Section: Introductionmentioning

confidence: 99%

Extend core UDF framework for GPU-enabled analytical query evaluation

Chen

Ren

Hsu

et al. 2011

Proceedings of the 15th Symposium on International Database Engineering &Amp; Applications - IDEAS '11

Self Cite

View full text Add to dashboard Cite

To achieve scalable data intensive analytics, we investigate methods to integrate general purpose analytic computation into a query pipeline using User Defined Functions (UDFs). However, an existing UDF cannot act as a block operator with chunk-wise input along the tuple-wise query processing pipeline, therefore unable to deal with the application semantics definable on the set of incoming tuples representing a single object or falling in a time window, and unable to leverage external computation engines for efficient batch processing.To enable the data intensive computation pipeline, we introduce a new kind of UDFs called Set-In Set-Out (SISO) UDFs. A SISO UDF is a block operator for processing the input tuples and returning the resulting tuples chunk by chunk. Operated in the query processing pipeline, a SISO UDF pools a chunk of input tuples, dispatches them to GPUs or an analytic engine in batch, materializes and then streams out the results. This behavior differentiates SISO UDF from all the existing ones, and makes efficient integration of analytic computation and data management feasible. We have implemented the SISO UDF framework by extending the PostgreSQL query engine, and further demonstrated the use of SISO UDF with GPU-enabled analytical query evaluation. Our experiments show that the proposed approach is scalable and efficient.

show abstract