Pipeline Parallelism for Inference on Heterogeneous Edge Computing

Hu, Yang; Imes, Connor; Zhao, Xuanang; Kundu, Souvik; Beerel, Peter A.; Crago, Stephen P.; Walters, John Paul

doi:10.48550/arxiv.2110.14895

Cited by 1 publication

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A model partitioning problem for model-distributed inference is designed using dynamic programming [5] with the cost of high complexity. As compared to [5], our AR-MDI algorithm (i) reduces the computation cost as our algorithm makes on the fly calculations while [5] solves a dynamic programming problem, (ii) makes model distribution fully decentralized while [5] relies on a genie to solve the dynamic programming problem and make the corresponding model distribution, (iii) is adaptive to timevarying resources, and (iv) recovers if there is a failure in the network. Model distributed inference is considered for a multi-source setup in [16].…”

Section: Related Workmentioning

confidence: 99%

“…The partial rate δ n−1 (k) corresponds to the rate in the denominator of (2), i.e., δ n−1 (k) = n−1 z=1 1/γ z (k). 5 On the other hand, (k−1) corresponds to the whole term, i.e., (k−1) = N z=1 1/γ z (k−1). Each worker calculates their partial rate by summing up the previous worker's partial rate and its own rate, i.e., ρ n (k)/d n (k) as shown in line 34.…”

Section: Ar-mdi: Adaptive and Resilient MDImentioning

confidence: 99%

“…We compare our algorithm AR-MDI with baselines; (i) EdgePipe [5], where model distribution is used for inference and the inference time is minimized as a solution of a dynamic programming problem, (ii) DDI, where data is distributed across workers, not the model itself, (iii) Local, where the source node has the complete DNN model and prefers to process all the data by itself, and (iv) Optimal, which is a solution to the optimization problem in (1). However, since the solution of ( 1) is computationally intensive, it incurs very high processing time in real implementation.…”

Section: A Datasets Dnn Models and Testbed Descriptionmentioning

confidence: 99%

“…It is crucial to exploit the potential of model-distributed inference in a heterogeneous and dynamic setup, where the computing power of workers may be different and change over time. A model partitioning mechanism based on dynamic programming is proposed in [5] for this purpose. However, this approach introduces too much computing cost to determine the optimal model allocation.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Adaptive and Resilient Model-Distributed Inference in Edge Computing Systems

Koyuncu

Seferoğlu

2023

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

The traditional approach to distributed deep neural network (DNN) inference in edge computing systems is data-distributed inference. In this paradigm, each worker has a pre-trained DNN model. Using the DNN model, the worker processes the data that is offloaded to itself. The data-distributed inference approach (i) has high communication cost especially when the size of data is large, and (ii) is not efficient in terms of memory as the whole model should be stored and computed in each worker. Model-distributed inference is emerging as a promising solution, where a DNN model is distributed across workers. Although there is a huge amount of work on model-distributed training, the benefit of model distribution for inference is not understood well. In this paper, we analyze the potential of model-distributed inference in edge computing systems. Then, we develop an Adaptive and Resilient Model-Distributed Inference (AR-MDI) algorithm based on our optimal model allocation formulation. AR-MDI performs model allocation in a lightweight and decentralized way and it is resilient against delayed workers and failures. We implement AR-MDI in a real testbed consisting of NVIDIA Jetson TX2s and show that AR-MDI improves the inference time significantly as compared to baselines when the size of data is large, such as ImageNet.INDEX TERMS Model-distributed inference, model parallelism, distributed deep neural networks, edge computing.

show abstract