CodHoop: A system for optimizing big data processing

Asad, Zakia; Chaudhry, Mohammad Asad Rehman; Malone, David

doi:10.1109/syscon.2015.7116767

Cited by 5 publications

(2 citation statements)

References 9 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Theorem 4 proves if the coding is performed at the network bisection then μ(bisection) ≥ 1 2 . We prove this corollary by presenting an example where μ < 1 2 when middlebox for performing coding is not placed at the network bisection, specifically in the example presented in Section B μ(bisection) = 1…”

Section: G Proof Of Corollarymentioning

confidence: 88%

“…The objective is to count the total number of instances when the smart meter with ID 1400 reported the power consumption exceeding a baseline of 0.5 for the following day and time codes: 11518, 35010, 00120, 20513. 1 A MapReduce task usually splits the input into independent chunks which are first processed by the mappers placed across different data center nodes in a completely parallel manner. The outputs of the mappers are then communicated to the reducers which are also placed across different nodes, during the shuffle phase, for further processing to complete the job.…”

Section: ) Hadoop Jobmentioning

confidence: 99%

See 1 more Smart Citation

Greener Data Exchange in the Cloud: A Coding-Based Optimization for Big Data Processing

Asad

Chaudhry

Malone

2016

IEEE J. Select. Areas Commun.

Self Cite

View full text Add to dashboard Cite

The rise of the cloud and distributed data-intensive (big data) applications puts pressure on data center networks due to the movement of massive volumes of data. Reducing the volume of communication is pivotal for embracing greener data exchange by efficient utilization of network resources. This paper proposes the use of mixing technique, spate coding, working in tandem with software-defined network control as a means of dynamicallycontrolled reduction in volume of communication. We introduce motivating real-world use-cases, and present a novel spate coding algorithm for the data center networks. We also analyze the computational complexity of the general problem of minimizing the volume of communication in a distributed data center application without degrading the rate of information exchange, and provide theoretical limits of such schemes. Moreover, we proceed to bridge the gap between theory and practice by performing a proof-of-concept implementation of the proposed system in a real world data center. We use Hadoop MapReduce, the most widely used big data processing framework, as our target. The experimental results employing two of industry standard benchmarks show the advantage of our proposed system compared to a vanilla Hadoop implementation, an in-network combiner, and Combine-N-Code. The proposed coding-based scheme shows performance improvement in terms of volume of communication (up to 62%), goodput (up to 76%), disk utilization (up to 38%), and the number of bits that can be transmitted per Joule of energy (up to 200%).

show abstract

Section: G Proof Of Corollarymentioning

confidence: 88%

Section: ) Hadoop Jobmentioning

confidence: 99%