Towards Scalable One-Pass Analytics Using MapReduce

Mazur, Edward J.; Li, Boduo; Diao, Yanlei; Shenoy, Prashant

doi:10.1109/ipdps.2011.251

Cited by 12 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, Thusoo et al discussed the framework adjustment when applying MapReduce to streaming, single‐pass processing. On the basis of this analysis, Mazur et al introduced the realization of a scalable platform of MapReduce for single‐pass analysis. In the streaming MapReduce process discussed in Li et al, who considered event stream processing; the Mapper and Reducer in MapReduce were redefined to improve the processing ability for continuous data.…”

Section: For Velocity: Real‐time Data‐centric System Architecturementioning

confidence: 99%

Process data analytics in the era of big data

Qin

2014

AIChE Journal

337

View full text Add to dashboard Cite

Section: For Velocity: Real‐time Data‐centric System Architecturementioning

confidence: 99%

Process data analytics in the era of big data

Qin

2014

AIChE Journal

337

View full text Add to dashboard Cite

“…Background on Incremental Hadoop. Our work is built on an improvement of Hadoop, called Incremental Hadoop [26,28]. As usual, an analytic task can be expressed as a number of rounds of map and reduce functions.…”

Section: System Designmentioning

confidence: 99%

Supporting scalable analytics with latency constraints

Diao

Shenoy

2015

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Recently there has been a significant interest in building big data analytics systems that can handle both "big data" and "fast data". Our work is strongly motivated by recent real-world use cases that point to the need for a general, unified data processing framework to support analytical queries with different latency requirements. Toward this goal, we start with an analysis of existing big data systems to understand the causes of high latency. We then propose an extended architecture with mini-batches as granularity for computation and shuffling, and augment it with new model-driven resource allocation and runtime scheduling techniques to meet user latency requirements while maximizing throughput. Results from real-world workloads show that our techniques, implemented in Incremental Hadoop, reduce its latency from tens of seconds to sub-second, with 2x-5x increase in throughput. Our system also outperforms state-ofthe-art distributed stream systems, Storm and Spark Streaming, by 1-2 orders of magnitude when combining latency and throughput.

show abstract

“…For example, Thusoo et al 50 discussed the framework adjustment when applying MapReduce to streaming, single-pass processing. On the basis of this analysis, Mazur et al 51 introduced the realization of a scalable platform of MapReduce for single-pass analysis. In the streaming MapReduce process discussed in Li et al, 52 who considered event stream processing; the Mapper and Reducer in MapReduce were redefined to improve the processing ability for continuous data.…”

Section: Mixed Modementioning

confidence: 99%

Process Data Analytics in the Era of Big Data

Qin

2014

AIChE J.

View full text Add to dashboard Cite

Towards Scalable One-Pass Analytics Using MapReduce

Cited by 12 publications

References 25 publications

Process data analytics in the era of big data

Process data analytics in the era of big data

Supporting scalable analytics with latency constraints

Process Data Analytics in the Era of Big Data

Contact Info

Product

Resources

About