Layerweaver: Maximizing Resource Utilization of Neural Processing Units via Layer-Wise Scheduling

Oh, Young Hoon; Kim, Seonghak; Jin, Yunho; Son, Sam; Bae, Jonghyun; Lee, Jongsung; Park, Yeonhong; Kim, Dong Uk; Ham, Tae Jun; Lee, Jae Woo

doi:10.1109/hpca51647.2021.00056

Cited by 28 publications

(10 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the wide spectrum of NPUs and DNN models, it is nearly impossible to balance the usage of NPU resources for all DNNs. Recently, several proposals have addressed this problem by time-multiplexing layer-wise execution of multiple DNN models with opposing characteristics (e.g., memory-intensive and compute-intensive) to saturate both compute and memory bandwidth [5], [12]. Figure 1 vidual requests.…”

Section: Limitations Of the Prior Artmentioning

confidence: 99%

“…Layerweaver+ replaces the greedy scheduler of Layerweaver [5] to achieve the two design goals: maximizing the inference throughput and bounding each request's latency to a given deadline. To this end, we introduce two operation modes in Layerweaver+: 1) Select the layer that causes the minimum resource idle time first, and 2) Select the layer with the minimum QoS slack time.…”

Section: Design and Implementationmentioning

confidence: 99%

“…The algorithm then finds the layer that incurs the minimum idle time to optimize throughput (Line 5-9) and temporarily updates the schedule state (Line 10). The Mode 1 scheduling is similar to Layerweaver [5] except for the case where multiple candidate layers of the same minimum idle time compete for selection. If there are such layers, the algorithm selects a layer from the request with the least QoS slack time as a tie-breaker.…”

Section: Design and Implementationmentioning

confidence: 99%

“…There exist some proposals that target to maximize throughput with latency-hiding techniques [4] or layer-wise time-multiplexed scheduling [5]. However, merely optimizing throughput is not sufficient for DNN inference as datacenters typically impose service-level agreement (SLA) goals [6] to set a bound for the maximum service latency.…”

Section: Introductionmentioning

confidence: 99%

“…• We identify the limitations of a state-of-the-art multitenant DNN scheduler (Layerweaver [5]), which does not take into account the QoS constraints of individual requests. • To address this, we propose Layerweaver+, which finds a more balanced schedule for both throughput and latency by switching between two scheduling modes: QoS-aware request prioritization and throughput-oriented scheduling.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Layerweaver+: A QoS-Aware Layer-Wise DNN Scheduler for Multi-Tenant Neural Processing Units

Jin

Ham

et al. 2022

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

Many cloud service providers employ specialized hardware accelerators, called neural processing units (NPUs), to accelerate deep neural networks (DNNs). An NPU scheduler is responsible for scheduling incoming user requests and required to satisfy the two, often conflicting, optimization goals: maximizing system throughput and satisfying quality-of-service (QoS) constraints (e.g., deadlines) of individual requests. We propose Layerweaver+, a low-cost layer-wise DNN scheduler for NPUs, which provides both high system throughput and minimal QoS violations. For a serving scenario based on the industry-standard MLPerf inference benchmark, Layerweaver+ significantly improves the system throughput by up to 266.7% over the baseline scheduler serving one DNN at a time.

show abstract

Section: Limitations Of the Prior Artmentioning

confidence: 99%

Section: Design and Implementationmentioning

confidence: 99%

Section: Design and Implementationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Layerweaver+: A QoS-Aware Layer-Wise DNN Scheduler for Multi-Tenant Neural Processing Units

Jin

Ham

et al. 2022

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures

Venieris

Panopoulos

Leontiadis

et al. 2021

2021 IEEE 32nd International Conference on Application-Specific Systems, Architectures and Processors (ASAP)

View full text Add to dashboard Cite

The unprecedented performance of deep neural networks (DNNs) has led to large strides in various Artificial Intelligence (AI) inference tasks, such as object and speech recognition. Nevertheless, deploying such AI models across commodity devices faces significant challenges: large computational cost, multiple performance objectives, hardware heterogeneity and a common need for high accuracy, together pose critical problems to the deployment of DNNs across the various embedded and mobile devices in the wild. As such, we have yet to witness the mainstream usage of state-of-the-art deep learning algorithms across consumer devices. In this paper, we provide preliminary answers to this potentially game-changing question by presenting an array of design techniques for efficient AI systems. We start by examining the major roadblocks when targeting both programmable processors and custom accelerators. Then, we present diverse methods for achieving real-time performance following a cross-stack approach. These span model-, system-and hardware-level techniques, and their combination. Our findings provide illustrative examples of AI systems that do not overburden mobile hardware, while also indicating how they can improve inference accuracy. Moreover, we showcase how custom ASIC-and FPGA-based accelerators can be an enabling factor for next-generation AI applications, such as multi-DNN systems. Collectively, these results highlight the critical need for further exploration as to how the various cross-stack solutions can be best combined in order to bring the latest advances in deep learning close to users, in a robust and efficient manner.

show abstract

TNPU: Supporting Trusted Execution with Tree-less Integrity Protection for Neural Processing Unit

Lee

Kim

et al. 2022

2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA)

View full text Add to dashboard Cite

Layerweaver: Maximizing Resource Utilization of Neural Processing Units via Layer-Wise Scheduling

Cited by 28 publications

References 39 publications

Layerweaver+: A QoS-Aware Layer-Wise DNN Scheduler for Multi-Tenant Neural Processing Units

Layerweaver+: A QoS-Aware Layer-Wise DNN Scheduler for Multi-Tenant Neural Processing Units

How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures

TNPU: Supporting Trusted Execution with Tree-less Integrity Protection for Neural Processing Unit

Contact Info

Product

Resources

About