AI Multi-Tenancy on Edge: Concurrent Deep Learning Model Executions and Dynamic Model Placements on Edge Devices

Subedi, Piyush; Hao, Jianwei; Kim, In Kee; Ramaswamy, Lakshmish

doi:10.1109/cloud53861.2021.00016

Cited by 16 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Only a few solutions have been designed for GPU management in the context of edge computing. For example, Subedi et al [39] mainly focuses on enabling GPU accelerated edge computation without considering latency-critical aspects such as placing applications close to the edge clients.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

NEPTUNE: Network- and GPU-aware Management of Serverless Functions at the Edge

Baresi¹,

Hu²,

Quattrocchi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Nowadays a wide range of applications is constrained by lowlatency requirements that cloud infrastructures cannot meet. Multiaccess Edge Computing (MEC) has been proposed as the reference architecture for executing applications closer to users and reducing latency, but new challenges arise: edge nodes are resourceconstrained, the workload can vary significantly since users are nomadic, and task complexity is increasing (e.g., machine learning inference). To overcome these problems, the paper presents NEP-TUNE, a serverless-based framework for managing complex MEC solutions. NEPTUNE i) places functions on edge nodes according to user locations, ii) avoids the saturation of single nodes, iii) exploits GPUs when available, and iv) allocates resources (CPU cores) dynamically to meet foreseen execution times. A prototype, built on top of K3S, was used to evaluate NEPTUNE on a set of experiments that demonstrate a significant reduction in terms of response time, network overhead, and resource consumption compared to three well-known approaches. CCS CONCEPTS• Theory of computation → Scheduling algorithms; • Computing methodologies → Distributed computing methodologies; • Computer systems organization → Distributed architectures.

show abstract

Section: Related Workmentioning

confidence: 99%

“…In addition, the tasks at the edge, like AI-based computations, are becoming heavier and heavier. The use of GPUs [9] can be fundamental to accelerate these computations, but they are seldom taken into account explicitly [6,20,39].…”

mentioning

confidence: 99%

NEPTUNE: Network- and GPU-aware Management of Serverless Functions at the Edge

Baresi¹,

Hu²,

Quattrocchi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…• We investigate the beneits and limitations of deploying DL models on edge devices with EdgeTPU clusters. This work is based on our preliminary version [65] and we take one step in a broader and more thorough evaluation of techniques for maximizing the DL inference throughput on edge devices and AI accelerators. In particular, to the best of our knowledge, this work is the irst study on evaluating DMP on heterogeneous edge resources/EdgeTPUs and characterizing the performance of EdgeTPU cluster for DL inference tasks.…”

Section: Introductionmentioning

confidence: 99%

Reaching for the Sky: Maximizing Deep Learning Inference Throughput on Edge Devices with AI Multi-Tenancy

Hao

Subedi

Ramaswamy

et al. 2023

ACM Trans. Internet Technol.

Self Cite

View full text Add to dashboard Cite

The wide adoption of smart devices and Internet-of-Things (IoT) sensors has led to massive growth in data generation at the edge of the Internet over the past decade. Intelligent real-time analysis of such a high volume of data, particularly leveraging highly accurate deep learning (DL) models, often requires the data to be processed as close to the data sources (or at the edge of the Internet) to minimize the network and processing latency. The advent of specialized, low-cost, and power-efficient edge devices has greatly facilitated DL inference tasks at the edge. However, limited research has been done to improve the inference throughput (e.g., number of inferences per second) by exploiting various system techniques. This study investigates system techniques, such as batched inferencing, AI multi-tenancy, and cluster of AI accelerators, which can significantly enhance the overall inference throughput on edge devices with DL models for image classification tasks. In particular, AI multi-tenancy enables collective utilization of edge devices’ system resources (CPU, GPU) and AI accelerators (e.g., Edge Tensor Processing Units; EdgeTPUs). The evaluation results show that batched inferencing results in more than 2.4 × throughput improvement on devices equipped with high-performance GPUs like Jetson Xavier NX. Moreover, with multi-tenancy approaches, e.g., concurrent model executions and dynamic model placements, the DL inference throughput on edge devices (with GPUs) and EdgeTPU can be further improved by up to 3 × and 10 ×, respectively. Furthermore, we present a detailed analysis of hardware and software factors that change the DL inference throughput on edge devices and EdgeTPUs, thereby shedding light on areas that could be further improved to achieve high-performance DL inference at the edge.

show abstract

“…The edge computing paradigm is emerging because the edge cluster can be deployed near data sources, e.g., IoT sensors, and process the data without relying on traditional data-center computing [4]- [6]. In particular, the development of small, energy-efficient, and capable CPU and AI accelerators for edge devices facilitates the adoption of edge computing to servicing various real-world applications, e.g., autonomous driving, drone-based surveillance systems, and environmental sensing [7]- [10].…”

Section: Introductionmentioning

confidence: 99%

“…IoT sensor and device layer ("bottom layer" in Fig. 1) has various IoT sensors and user devices, which perform diverse sensing operations, and user devices (including edge devices) are used to host lightweight applications to perform real-time on-board processing, including sensing data filtering and noise removal [53], lightweight AI inference tasks [10], [54], and stream processing [55].…”

Section: Introductionmentioning

confidence: 99%

EdgeFaaSBench: Benchmarking Edge Devices Using Serverless Computing

Rajput

Kulkarni

Cho

et al. 2022

2022 IEEE International Conference on Edge Computing and Communications (EDGE)

Self Cite

View full text Add to dashboard Cite

Due to the development of small-size, energyefficient, and powerful CPUs and GPUs for single board computers, various edge devices are widely adopted for hosting real-world applications, including real-time object detection, autonomous driving, and sensor stream processing. At the same time, serverless computing receives increasing attention as a new application deployment model because of its simplicity, scalability, event-driven processing, and short-lived computation. Therefore, there is a growing demand for applying serverless computing to edge computing environments. However, due to the lack of characterization of serverless edge computing (e.g., application performance and impact from resource heterogeneity), researchers and practitioners have to conduct tedious measurements to understand the performance of serverless applications on edge devices in non-systematic ways.We create EdgeFaaSBench, a novel benchmark suite for serverless computing on edge devices, to bridge this gap. EdgeFaaSBench is developed on top of Apache OpenFaaS with Docker Swarm and can run various serverless benchmark workloads on edge devices with different hardware specifications (e.g., GPUs). EdgeFaaSBench contains 14 different benchmark workloads running on heterogeneous edge devices and captures various system-level, application-level, and serverless-specific metrics, including system utilization, response time, cold/warm start times, and impact of concurrent function executions. Experimental studies are conducted on two widely used edge devices, Raspberry Pi 4B and Jetson Nano, to show EdgeFaaSBench's capabilities to benchmark serverless computing on edge devices.

show abstract

AI Multi-Tenancy on Edge: Concurrent Deep Learning Model Executions and Dynamic Model Placements on Edge Devices

Cited by 16 publications

References 20 publications

NEPTUNE: Network- and GPU-aware Management of Serverless Functions at the Edge

NEPTUNE: Network- and GPU-aware Management of Serverless Functions at the Edge

Reaching for the Sky: Maximizing Deep Learning Inference Throughput on Edge Devices with AI Multi-Tenancy

EdgeFaaSBench: Benchmarking Edge Devices Using Serverless Computing

Contact Info

Product

Resources

About