E-BATCH: Energy-Efficient and High-Throughput RNN Batching

Abstract

Recurrent Neural Network (RNN) inference exhibits low hardware utilization due to the strict data dependencies across time-steps. Batching multiple requests can increase throughput. However, RNN batching requires a large amount of padding since the batched input sequences may vastly differ in length. Schemes that dynamically update the batch every few time-steps avoid padding. However, they require executing different RNN layers in a short time span, decreasing energy efficiency. Hence, we propose E-BATCH, a low-latency and energy-efficient batching scheme tailored to RNN accelerators. It consists of a runtime system and effective hardware support. The runtime concatenates multiple sequences to create large batches, resulting in substantial energy savings. Furthermore, the accelerator notifies it when the evaluation of an input sequence is done. Hence, a new input sequence can be immediately added to a batch, thus largely reducing the amount of padding. E-BATCH dynamically controls the number of time-steps evaluated per batch to achieve the best trade-off between latency and energy efficiency for the given hardware platform. We evaluate E-BATCH on top of E-PUR and TPU. E-BATCH improves throughput by 1.8× and energy efficiency by 3.6× in E-PUR, whereas in TPU, it improves throughput by 2.1× and energy efficiency by 1.6×, over the state-of-the-art.

SHARP: An Adaptable, Energy-Efficient Accelerator for Recurrent Neural Networks

“…As depicted, each LSTM gate performs two matrix-vector multiplications (MVMs), which finally decide how to update the cell-state (c t ), and how to generate the hidden output vector (h t ) that is recurrently sent to the following time step. Two kinds of dependencies exist in these The figure shows E-PUR's [21] speedup running EESEN [8], for a range of MAC units. Due to the scalability issue, it does not achieve performance improvement proportional to the increase in resources.…”

“…For instance, NPUs [24,28] have the parallel multiply-accumulation (MAC) stage as the heart of their pipeline and are not optimized in case the serial part becomes the performance bottleneck for some models. On the other hand, customized accelerators [21,29] use a relatively small resource budget which therefore causes large delay for MVM, hence overlap the remaining LSTM computation that needs to run sequentially. However, when using more MACs, the issue of efficiently handling LSTM's dependencies still remains.…”

“…To meet the requirements of real-time inference at large scale, a high-performance and energy efficient accelerator for RNN is highly desired. However, two reasons make it very difficult to accomplish efficient RNN computation by CPUs or GPUs in parallel [21,22]: (1) recurrent behaviour of RNN architecture which imposes several data-dependencies, (2) limited parallel tasks due to the enforced low batch size by Service-Level Agreements (SLAs) in the inference evaluation [23,24]. FLOP Efficiency (%)…”

“…Because the performance of RNN on general-purpose processors has difficulties to meet the requirements of realtime inference in modern applications, accelerating RNN through either customized architectures [21,27] or neural processing units (NPU) [24,28] has been recently explored. These systems are implemented on either ASICs or FPGAs.…”

Complex Theory and Batch Processing in Mechanical Systemic Data Extraction

“…Meanwhile, based on the learning level set of the mean simulator response, several new schemes were developed, including Multi-level Batching (MLB), Ratchet Batching (RB), Adaptive Batched Stepwise Uncertainty Reduction (ABSUR), Adaptive Design with the Stepwise Allocation (ADSA), and Deterministic Design with the Stepwise Allocation (DDSA). MLB, RB, and ABSUR simultaneously or ADSA and DDSA sequentially determine the sequential design inputs and the respective number of replicates [3]. By Bermudan option pricing via Monte Carlo regression, quantitative applications in many financial instances showed that the method has a significant computational speed and a low distortion rate.…”