Adaptive and Collaborative Edge Inference in Task Stream with Latency Constraint

Song, Jinduo; Liu, Zhicheng; Wang, Xiaofei; Qiu, Chao; Chen, Xu

doi:10.1109/icc42927.2021.9500892

Cited by 10 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For tasks with deadlines, edgent can maximize accuracy without exceeding the deadline. Song et al [49] also propose a collaborative inference system based on edgent, which applies early-exit mechanism and model division technology to solve the problem of EI in task flow scenarios. At the same time, the authors design an offline dynamic programming algorithm and an online deep reinforcement learning algorithm to dynamically select the exit point and partition point of the model in the task flow, so as to balance the efficiency and accuracy of inference tasks.…”

Section: Inference Accuracy Maximizationmentioning

confidence: 99%

A Survey on Collaborative DNN Inference for Edge Intelligence

Ren¹,

Qu²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the vigorous development of artificial intelligence (AI), the intelligent applications based on deep neural network (DNN) change people's lifestyles and the production efficiency. However, the huge amount of computation and data generated from the network edge becomes the major bottleneck, and traditional cloud-based computing mode has been unable to meet the requirements of real-time processing tasks. To solve the above problems, by embedding AI model training and inference capabilities into the network edge, edge intelligence (EI) becomes a cutting-edge direction in the field of AI. Furthermore, collaborative DNN inference among the cloud, edge, and end device provides a promising way to boost the EI. Nevertheless, at present, EI oriented collaborative DNN inference is still in its early stage, lacking a systematic classification and discussion of existing research efforts. Thus motivated, we have made a comprehensive investigation on the recent studies about EI oriented collaborative DNN inference. In this paper, we firstly review the background and motivation of EI. Then, we classify four typical collaborative DNN inference paradigms for EI, and analyze the characteristics and key technologies of them. Finally, we summarize the current challenges of collaborative DNN inference, discuss the future development trend and provide the future research direction.

show abstract

Section: Inference Accuracy Maximizationmentioning

confidence: 99%

A Survey on Collaborative DNN Inference for Edge Intelligence

Ren¹,

Qu²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, given S i,i , step 4 can provide S i,j , since the grouping policy that gives S i,i is the same as that of S i,j except the last group. To derive S i+1,i+1 , we first need to derive the feasible region of the second last group, where the latency constraint of the second last group cannot be too close to that of the last group according to (20). As a result, D is used to denote the set of all feasible indices of the first task in the second last group (step 6).…”

Section: B Different Latency Constraintsmentioning

confidence: 99%

“…The authors minimize the coinference latency under light workload scenarios or maximize the co-inference throughput under heavy workload scenarios. Further, the authors of [20] extend Edgent to the scenarios with random task arrivals. They propose a deep reinforcement learning agent to balance the trade-off between the number of tasks completed within the latency constraints and the inference accuracy.…”

Section: Introductionmentioning

confidence: 99%

“…Nevertheless, the edge server processes the offloaded tasks in a first-in-first-out (FIFO) manner in [19], [20]. To the best of our knowledge, only [21] considers multi-user co-inference, wherein the DNN partitioning and edge server CPU cores allocation are jointly optimized to minimize the maximum inference latency among users.…”

Section: Introductionmentioning

confidence: 99%

“…n (|G i+1 |) is the maximum length of the edge occupancy period of G i+1 . Although assumption(20) restricts certain grouping policies, it can greatly reduce the complexity by forcing the tasks with similar latency constraints to be grouped together, without degrading the performance too much 3. Alg.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Multi-user Co-inference with Batch Processing Capable Edge Server

Shi¹,

Zhou²,

Niu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graphics processing units (GPUs) can improve deep neural network inference throughput via batch processing, where multiple tasks are concurrently processed. We focus on novel scenarios that the energy-constrained mobile devices offload inference tasks to an edge server with GPU. The inference task is partitioned into sub-tasks for a finer granularity of offloading and scheduling, and the user energy consumption minimization problem under inference latency constraints is investigated. To deal with the coupled offloading and scheduling introduced by concurrent batch processing, we first consider an offline problem with a constant edge inference latency and the same latency constraint. It is proven that optimizing the offloading policy of each user independently and aggregating all the same sub-tasks in one batch is optimal, and thus the independent partitioning and same sub-task aggregating (IP-SSA) algorithm is inspired. Further, the optimal grouping (OG) algorithm is proposed to optimally group tasks when the latency constraints are different. Finally, when future task arrivals cannot be precisely predicted, a deep deterministic policy gradient (DDPG) agent is trained to call OG. Experiments show that IP-SSA reduces up to 94.9% user energy consumption in the offline setting, while DDPG-OG outperforms DDPG-IP-SSA by up to 8.92% in the online setting.

show abstract

A Survey on Collaborative DNN Inference for Edge Intelligence

Ren

Dong

et al. 2023

Mach. Intell. Res.

View full text Add to dashboard Cite

Adaptive and Collaborative Edge Inference in Task Stream with Latency Constraint

Cited by 10 publications

References 13 publications

A Survey on Collaborative DNN Inference for Edge Intelligence

A Survey on Collaborative DNN Inference for Edge Intelligence

Multi-user Co-inference with Batch Processing Capable Edge Server

A Survey on Collaborative DNN Inference for Edge Intelligence

Contact Info

Product

Resources

About