DNN Partitioning for Inference Throughput Acceleration at the Edge

Abstract

Deep neural network (DNN) inference on streaming data requires computing resources to satisfy inference throughput requirements. However, latency and privacy sensitive deep learning applications cannot afford to offload computation to remote clouds because of the implied transmission cost and lack of trust in third-party cloud providers. Among solutions to increase performance while keeping computation on a constrained environment, hardware acceleration can be onerous, and model optimization requires extensive design efforts while hindering accuracy. DNN partitioning is a third complementary approach, and consists of distributing the inference workload over several available edge devices, taking into account the edge network properties and the DNN structure, with the objective of maximizing the inference throughput (number of inferences per second). This paper introduces a method to predict inference and transmission latencies for multi-threaded distributed DNN deployments, and defines an optimization process to maximize the inference throughput. A branch and bound solver is then presented and analyzed to quantify the achieved performance and complexity. This analysis has led to the definition of the acceleration region, which describes deterministic conditions on the DNN and network properties under which DNN partitioning is beneficial. Finally, experimental results confirm the simulations and show inference throughput improvements in sample edge deployments.

On the Dynamics of Single-Orbit Requester-Provider Systems

“…is paper focuses on the theoretical and experimental study of stability in distributed, decentralized systems for resource discovery and allocation in non-deterministic, heterogenous, time-variant networking and communication infrastructures. is analysis can be relevant for a number of use cases: multicast content distribution in data centers [1] or in other constrained infrastructures, distributed applications and systems including information-centric architectures [2], mechanisms for dynamic allocation of workloads in the cloud [3], computing resources in data centers [4] or in other deployments of the Internet edge [5]. In all these systems, the study of performance is challenging as these systems increasingly rely on distributed multi-party architectures, involving uncoordinated or loosely coordinated agents that interact to each other, may have partial, possibly inconsistent views of the environment, and follow autonomous, possibly con icting policies.…”

“…Fig. 6 displays the Beta ing 5 distribution parameters for N = 500 samples of single-orbit (b-orbit) processes at iteration n = 3000, a er excluding extreme values (in the case of b-orbit, zeros 6 ).…”