Recovery-oriented Big Data Computing for Exactly Once Message Processing

Sun, Fangchen; Suo, Xiaotong; Kamat, Nishad; Mao, Feng; Guo, Stephen; Yao, Yiyang; Malaviya, Paritosh; Bhatt, Kushal; Gupta, Anurag; Jain, Mridul; Achan, Kannan

doi:10.1109/bigdata47090.2019.9006585

Cited by 1 publication

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The solution proposed in [21] improves the message queue systems, e.g. Kafka [22] and RabbitMQ [23], to ensure exactly once processing through a consumer side protocol.…”

Section: A Collaborative Edge In Iotmentioning

confidence: 99%

ECE: Exactly Once Computation for Collaborative Edge in IoT Using Information-Centric Networking

Wang,

Lee,

Murray

et al. 2023

IEEE Internet Things J.

View full text Add to dashboard Cite

Exactly-once data processing/delivery can be guaranteed in traditional big data processing systems, e.g. Apache Flink. Checkpoint is commonly used as the solution. Each operator in these systems can restart from the last successfully saved state whenever a failure happens. It is not necessary to restore the logical job graph onto the same device(s) in traditional datacentre scenarios with powerful servers close to each other. However, the datacentre oriented solutions are not suitable for IoT collaborative edge computing scenarios. The logical job graph is tightly coupled to the physical topology in IoT networks. Data processing task(s) cannot be placed at a random edge device to recover from a network failure as it needs to evaluate the benefits of transmitting data versus processing/aggregating the data. To address the above challenges, this paper proposes an Information Centric Networking based solution and correspondent protocols to provide Exactly-oncecomputation for the Collaborative Edge in IoT (ECE). It contains a job execution scheme to deliver IoT jobs with exactly once data computation guarantee and a recovery procedure to dynamically change the IoT job execution graph while experiencing link failures. The protocol also provides a checking procedure on data state (received/un-received and computed/un-computed) to prevent any data loss or duplicated data processing due to the updated job graph. A data identification approach based on the job graph is devised to support the ECE functionality. A testbed has been developed on ndnSIM and the simulation results have verified the feasibility and scalability of ECE design. It also evaluates the overhead incurred by the ECE protocol to guarantee exactly once data computation.

show abstract

“…The solution proposed in [21] improves the message queue systems, e.g. Kafka [22] and RabbitMQ [23], to ensure exactly once processing through a consumer side protocol.…”

Section: A Collaborative Edge In Iotmentioning

confidence: 99%