Multi-DNN Accelerators for Next-Generation AI Systems

Abstract

As the use of AI-powered applications widens across multiple domains, so do increase the computational demands. Primary driver of AI technology are the deep neural networks (DNNs). When focusing either on cloud-based systems that serve multiple AI queries from different users each with their own DNN model, or on mobile robots and smartphones employing pipelines of various models or parallel DNNs for the concurrent processing of multi-modal data, the next generation of AI systems will have multi-DNN workloads at their core. Large-scale deployment of AI services and integration across mobile and embedded systems require additional breakthroughs in the computer architecture front, with processors that can maintain high performance as the number of DNNs increases while meeting the quality-of-service requirements, giving rise to the topic of multi-DNN accelerator design.

HASP: Hierarchical Asynchronous Parallelism for Multi-NN Tasks

“…The tile-grained scheduler design for multi-NN in data centers enables the hardware to handle multi-tenant requests, but introduces additional data movement and scheduling latency in mobile machines with long-term multi-NN deployment and high-throughput processing requirement, resulting unstable system performance. Few studies address accelerated multi-NN on mobile and edge devices but with customized architectures, restricted programmability, and limited scalability [13]. A programmable and high performance multi-NN processor for mobile and edge computing is still lacking.…”

“…The advantages of spatial multi-tasking over temporal are further explored in the Section 2. However, as listed in Table 1, multiple systolic arrays based accelerator [10] and customized accelerator [15] can not well satisfy the scale-out requirements for the inflexible dataflow pattern, and heterogeneous accelerators limit the level of applied hardware customization [13].…”

HASP: Hierarchical Asynchronous Parallelism for Multi-NN Tasks

“…The tile-grained scheduler design for multi-NN in data centers enables the hardware to handle multi-tenant requests, but introduces additional data movement and scheduling latency in mobile machines with long-term multi-NN deployment and high-throughput processing requirement, resulting unstable system performance. Few studies address accelerated multi-NN on mobile and edge devices but with customized architectures, restricted programmability, and limited scalability [13]. A programmable and high performance multi-NN processor for mobile and edge computing is still lacking.…”

“…The advantages of spatial multi-tasking over temporal are further explored in the Section 2. However, as listed in Table 1, multiple systolic arrays based accelerator [10] and customized accelerator [15] can not well satisfy the scale-out requirements for the inflexible dataflow pattern, and heterogeneous accelerators limit the level of applied hardware customization [13].…”