A 1ynm 1.25V 8Gb 16Gb/s/Pin GDDR6-Based Accelerator-in-Memory Supporting 1TFLOPS MAC Operation and Various Activation Functions for Deep Learning Application

Kwon, Daehan; Lee, Seongju; Kim, Kyuyoung; Oh, Sanghoon; Park, Joonhong; Hong, Gi-Moon; Ka, Dongyoon; Hwang, Kyu-Dong; Park, Jeongje; Kang, Kyeong-Pil; Kim, Jungyeon; Jeon, Junyeol; Kim, Nahsung; Kwon, Yongkee; Vladimir, Kornijcuk; Shin, Woojae; Won, Jongsoon; Lee, Min-Kyu; Joo, Hyunha; Choi, Haerang; Kim, Guhyun; An, Byeongju; Lee, Jae‐Wook; Ko, Donguc; Jun, Younggun; Kim, Ilwoong; Song, Choungki; Kim, Il Kon; Park, Chan-Wook; Kim, Se‐Ho; Jeong, Chunseok; Lim, Euicheol; Kim, Dongkyun; Jang, Jieun; Park, Il; Chun, Junhyun; Cho, Joohwan

doi:10.1109/jssc.2022.3200718

Cited by 9 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For normal DRAM commands, we adopt GDDR5 timing constrains in 39 to make a conservative estimation due to the lack of detailed information of GDDR6. Similarly, the current values are obtained from DDR5 datasheet 40 and multiplied by 3 to account for the current consumption increase during parallel bank operations 27 . The GDDR6 I/O access energy 5.5 pJ/bit is adopted from GDDR6 datasheet 41 .…”

Section: Evaluation Methodsmentioning

confidence: 99%

PIM-GPT: A Hybrid Process-in-Memory Accelerator for Autoregressive Transformers

Wu,

Wang,

2024

Preprint

View full text Add to dashboard Cite

Decoder-only Transformer models such as GPT have demonstrated exceptional performance in text generation, by autore-gressively predicting the next token. However, the efficacy of running GPT on current hardware systems is bounded by low compute-to-memory-ratio and high memory access. Process-in-memory (PIM) architectures can minimize off-chip data movement and utilize high internal bandwidth. They stand out as promising candidates for accelerating memory-bounded tasks such as GPT inference. In this work, we propose a PIM accelerator, PIM-GPT, which achieves end-to-end acceleration of GPT inference with high performance and high energy efficiency. PIM-GPT leverages DRAM-based PIM designs for executing multiply-accumulate (MAC) operations directly in the DRAM chips, eliminating the need to move matrix data off-chip. Non-linear functions and data communication is supported by an application specific integrated chip (ASIC). At the software level, mapping schemes are designed to maximize data locality and computation parallelism by concatenating and partitioning matrices among DRAM channels and banks to utilize all available in-memory computation units. The efficiency of the PIM-GPT architecture is verified through circuit synthesis and an event-driven clock-cycle accurate simulator. Overall, PIM-GPT achieves 41−137×, 631−1074× speedup and 123−383×, 320−602× energy efficiency over GPU and CPU baseline on 8 GPT models with up to 1.4 billion parameters.

show abstract

Section: Evaluation Methodsmentioning

confidence: 99%

PIM-GPT: A Hybrid Process-in-Memory Accelerator for Autoregressive Transformers

Wu,

Wang,

2024

Preprint

View full text Add to dashboard Cite

show abstract

PIM GPT a hybrid process in memory accelerator for autoregressive transformers

Wu,

Wang,

2024

npj Unconv. Comput.

View full text Add to dashboard Cite

Decoder-only Transformer models such as Generative Pre-trained Transformers (GPT) have demonstrated exceptional performance in text generation by autoregressively predicting the next token. However, the efficiency of running GPT on current hardware systems is bounded by low compute-to-memory-ratio and high memory access. In this work, we propose a Process-in-memory (PIM) GPT accelerator, PIM-GPT, which achieves end-to-end acceleration of GPT inference with high performance and high energy efficiency. PIM-GPT leverages DRAM-based PIM designs for executing multiply-accumulate (MAC) operations directly in the DRAM chips, eliminating the need to move matrix data off-chip. Non-linear functions and data communication are supported by an application specific integrated chip (ASIC). At the software level, mapping schemes are designed to maximize data locality and computation parallelism. Overall, PIM-GPT achieves 41 − 137 × , 631 − 1074 × speedup and 123 − 383 × , 320 − 602 × energy efficiency over GPU and CPU baseline on 8 GPT models with up to 1.4 billion parameters.

show abstract