GRNN

Abstract

Recurrent neural networks (RNNs) have gained significant attention due to their effectiveness in modeling sequential data, such as text and voice signal. However, due to the complex data dependencies and limited parallelism, current inference libraries for RNNs on GPUs produce either high latency or poor scalability, leading to inefficient resource utilization. Consequently, companies like Microsoft and Facebook use CPUs to serve RNN models.This work demonstrates the root causes of the unsatisfactory performance of existing implementations for RNN inference on GPUs from several aspects, including poor data reuse, low on-chip resource utilization, and high synchronization overhead. We systematically address these issues and develop a GPU-based RNN inference library, called GRNN, that provides low latency, high throughput, and efficient resource utilization. GRNN minimizes global memory accesses and synchronization overhead, as well as balancing on-chip resource usage through novel data reorganization, thread mapping, and performance modeling techniques. Evaluated on extensive benchmarking and real-world applications, we show that GRNN outperforms the state-of-the-art CPU inference library by up to 17.5X and state-of-the-art GPU inference libraries by up to 9X in terms of latency reduction.

SHARP: An Adaptable, Energy-Efficient Accelerator for Recurrent Neural Networks

“…We compare SHARP against the state-of-the-art GPU, FPGA and ASIC implementations, i.e. cuDNN [20], GRNN [23],…”

“…To meet the requirements of real-time inference at large scale, a high-performance and energy efficient accelerator for RNN is highly desired. However, two reasons make it very difficult to accomplish efficient RNN computation by CPUs or GPUs in parallel [21,22]: (1) recurrent behaviour of RNN architecture which imposes several data-dependencies, (2) limited parallel tasks due to the enforced low batch size by Service-Level Agreements (SLAs) in the inference evaluation [23,24]. FLOP Efficiency (%)…”

“…Second, for the online inference scenario, queries come in one-by-one and have stringent latency SLA, often in single milliseconds [23,24]. This requirement further reduces data reuse and available parallelism in RNN inference.…”

“…This requirement further reduces data reuse and available parallelism in RNN inference. Recently, there have been several efforts on either CPU [26], or GPU [23,22], to improve the efficiency of RNN inference. However, they show poor scalibility for either small or large models with different sequence length.…”

CAGE: Complementing Arm CCA with GPU Extensions

“…While CCA provides strong data security and enables confidential computing on next-generation Arm devices, the support for GPUs [4], [21], [69], which are widely used to accelerate the general-, high-performance, and artificial intelligence computing scenarios [15], [18], [34], [45], [55], is only recently proposed. However, such support, called RME Device Assignment (RME-DA) [25], is currently a high-level concept without completed hardware implementation.…”