DREW: Efficient Winograd CNN Inference with Deep Reuse

Wu, Ruofan; Zhang, Feng; Guan, Jiawei; Zheng, Zishan; Du, Xiaoyong; Shen, Xipeng

doi:10.1145/3485447.3511985

Cited by 9 publications

(2 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xie et al [47] proposed to build an efficient two-phase LSM tree as a lookup table by leveraging the capacity advantages of Intel Optane Persistent Memory to speed up the Molecular Dynamic (MD) simulation. Some studies [29,45,28] empirically proved that a similar computation exists in CNN training and inference and proposed to project similar CNN computation results into buckets by locality-sensitive hashing (LSH) to speed up the CNN training and inference.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

MEMO : Accelerating Transformers with Memoization on Big Memory Systems

Feng¹,

Jeon²,

Blagojević³

et al. 2023

Preprint

View full text Add to dashboard Cite

Transformers gain popularity because of their superior prediction accuracy and inference throughput. However, the transformer is computation-intensive, causing a long inference time. The existing work to accelerate transformer inferences has limitations because of the changes to transformer architectures or the need for specialized hardware. In this paper, we identify the opportunities of using memoization to accelerate the attention mechanism in transformers without the above limitation. Built upon a unique observation that there is a rich similarity in attention computation across inference sequences, we build an attention database upon the emerging big memory system. We introduce the embedding technique to find semantically similar inputs to identify computation similarity. We also introduce a series of techniques such as memory mapping and selective memoization to avoid memory copy and unnecessary overhead. We enable 21% performance improvement on average (up to 68%) with the TB-scale attention database and with ignorable loss in inference accuracy.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Similarity in DNNs Prior works in [29,28,45,35] report that high similarity exists in convolution computation in a single image data or RNN neuron activation in a consecutive time step. Cao et al [4] proposed a decoupled transformer architecture dedicated to Question-Answering (QA) tasks.…”

Section: Related Workmentioning

confidence: 99%