A Survey on Collaborative DNN Inference for Edge Intelligence

Abstract

With the vigorous development of artificial intelligence (AI), the intelligent applications based on deep neural network (DNN) change people's lifestyles and the production efficiency. However, the huge amount of computation and data generated from the network edge becomes the major bottleneck, and traditional cloud-based computing mode has been unable to meet the requirements of real-time processing tasks. To solve the above problems, by embedding AI model training and inference capabilities into the network edge, edge intelligence (EI) becomes a cutting-edge direction in the field of AI. Furthermore, collaborative DNN inference among the cloud, edge, and end device provides a promising way to boost the EI. Nevertheless, at present, EI oriented collaborative DNN inference is still in its early stage, lacking a systematic classification and discussion of existing research efforts. Thus motivated, we have made a comprehensive investigation on the recent studies about EI oriented collaborative DNN inference. In this paper, we firstly review the background and motivation of EI. Then, we classify four typical collaborative DNN inference paradigms for EI, and analyze the characteristics and key technologies of them. Finally, we summarize the current challenges of collaborative DNN inference, discuss the future development trend and provide the future research direction.

Network for Distributed Intelligence: a Survey and Future Perspectives

“…Collaboration among devices in proximity is also beneficially exploited for inference splitting in [84], where close devices exchange data via the Web real-time communication protocol. Similar approaches are surveyed in [99] and references therein.…”

Balanced segmentation of CNNs for multi-TPU inference

“…The balancedSplit function implements a binary search that calls at each step to the greedy method splitCheck to verify if the array P can be segmented into s parts with at most bound parameters in each. This method traverses the array assigning values to a fragment as long as their sum is less than bound, and starting with a new fragment when exceeding it (lines [19][20][21][22][23][24]. If the final amount of fragments needed does not exceed the required s, then bound is a sufficient upper bound and smaller values are searched (lines 8-10).…”

“…Edge Computing aims at bringing computations near to the sensors in Internet of Things (IoT) deployments, in order to improve latencies, increase security and reduce access cost to datacenters. The convergence of Edge Computing and Artificial Intelligence (AI) tasks in the so-called Edge-AI paradigm [14,20] pursues bringing intelligence to edge devices in order to cover a number of applications necessary in many IoT scenarios (object detection for smart cameras [9], smart city applications [24], healthcare [1] or autonomous driving [16]), among others), bringing all the benefits of Edge Computing to the AI arena.…”

A Relay-Assisted Communication Scheme for Collaborative On-Device CNN Execution Considering Hybrid Parallelism

“…The common solution of EI is to deploy powerful edge servers to operate DL execution at the edge. In scenarios where deploying these servers is challenging, costly, or even impossible in high-mobility and harsh scenarios, device-to-device collaborative DL execution could be a viable approach to perform these applications [19].…”