9.1 A 7nm 4-Core AI Chip with 25.6TFLOPS Hybrid FP8 Training, 102.4TOPS INT4 Inference and Workload-Aware Throttling

Agrawal, Ankur; Lee, Sae Kyu; Silberman, J. A.; Ziegler, Matthew M.; Kang, Mingu; Venkataramani, Swagath; Cao, Nianwen; Fleischer, Bruce; Guillorn, Michael A.; Cohen, Matthew; Mueller, Silvia Melitta; Oh, Jinwook; Lutz, Martin; Jung, Jinwook; Koswatta, Siyu; Zhou, Ching; Zalani, Vidhi; Bonanno, James; Casatuta, Robert; Chen, Chia-Yu; Choi, Jungwook; Haynie, Howard; Herbert, Alyssa; Jain, Radhika; Kar, Monodeep; Kim, Kyu Hyun; Li, Yulong; Ren, Zhibin; Rider, Scot; Schaal, Marcel; Schelm, Kerstin; Scheuermann, M.; Sun, Xiao; Tran, Hung; Wang, Naigang; Wang, Wei; Zhang, Xin; Shah, Vinay; Curran, Brian; Srinivasan, Vijayalakshmi; Lu, Pong‐Fei; Shukla, Sunil; Chang, Leland; Gopalakrishnan, Kailash

doi:10.1109/isscc42613.2021.9365791

Cited by 61 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The computation model of a neuron is illustrated in Fig. 2 and given by (1). Each neuron performs a weighted sum of all its inputs and then a bias term is added for a possible offset [12].…”

Section: Brief Background On Deep Neural Networkmentioning

confidence: 99%

“…In [114] fundamental bit decomposition architectures (vertical and horizontal decomposition) are further approximated by constraining the maximum value of the partial sums. IBM RAPID [30] uses DLFloat16, 2-bit (INT2) and 1-bit fxp while [1] supports DLFloat16 and HFP8 formats as well as INT4 and INT2 formats for highly scaled inference. Intel Spring Hill [129] supports FP16 as well as INT8, INT4, INT2, and even 1 bit precision operations natively.…”

Section: Precision Scalingmentioning

confidence: 99%

“…Considering the high demand for edge AI [45], the billions of mobile devices running DNN inference, and the rapid growth of AI chips 1 , our focus in this survey is to study, analyze, and elucidate the impact of hardware approximation techniques on the efficiency and accuracy of DNN inference accelerators. Prior research on DNN accelerators reports that between 30% to 80% of the system energy is consumed by DRAM [63] with data movement dominating the energy consumption [130].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

et al. 2022

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are very popular because of their high performance in various cognitive tasks in Machine Learning (ML). Recent advancements in DNNs have brought beyond human accuracy in many tasks, but at the cost of high computational complexity. To enable efficient execution of DNN inference, more and more research works, therefore, exploit the inherent error resilience of DNNs and employ Approximate Computing (AC) principles to address the elevated energy demands of DNN accelerators. This article provides a comprehensive survey and analysis of hardware approximation techniques for DNN accelerators. First, we analyze the state of the art and by identifying approximation families, we cluster the respective works with respect to the approximation type. Next, we analyze the complexity of the performed evaluations (with respect to the dataset and DNN size) to assess the efficiency, the potential, and limitations of approximate DNN accelerators. Moreover, a broad discussion is provided, regarding error metrics that are more suitable for designing approximate units for DNN accelerators as well as accuracy recovery approaches that are tailored to DNN inference. Finally, we present how Approximate Computing for DNN accelerators can go beyond energy efficiency and address reliability and security issues, as well.

show abstract

Section: Brief Background On Deep Neural Networkmentioning

confidence: 99%

Section: Precision Scalingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

et al. 2022

View full text Add to dashboard Cite

show abstract

“…We consider a system with a CPU attached to a co-processor capable of executing common deep learning operations such as GEMM, LSTM activation etc. at various data precisions [30]. The co-processor executes the compute-heavy data-parallel portions of the workload, while control-heavy operations (e.g., sorting and pruning hypothesis) are mapped to the CPU.…”

Section: Inference Performance In End-to-end Modelsmentioning

confidence: 99%

4-bit Quantization of LSTM-based Speech Recognition Models

Fasoli¹,

Chen²,

Serrano³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We investigate the impact of aggressive low-precision representations of weights and activations in two families of large LSTM-based architectures for Automatic Speech Recognition (ASR): hybrid Deep Bidirectional LSTM -Hidden Markov Models (DBLSTM-HMMs) and Recurrent Neural Network -Transducers (RNN-Ts). Using a 4-bit integer representation, a naïve quantization approach applied to the LSTM portion of these models results in significant Word Error Rate (WER) degradation. On the other hand, we show that minimal accuracy loss is achievable with an appropriate choice of quantizers and initializations. In particular, we customize quantization schemes depending on the local properties of the network, improving recognition performance while limiting computational time. We demonstrate our solution on the Switchboard (SWB) and CallHome (CH) test sets of the NIST Hub5-2000 evaluation. DBLSTM-HMMs trained with 300 or 2000 hours of SWB data achieves <0.5% and <1% average WER degradation, respectively. On the more challenging RNN-T models, our quantization strategy limits degradation in 4-bit inference to 1.3%.

show abstract

“…15 In early 2021, IBM announced the results of its work on the world's first energy-efficient AI chip with low-precision training and inference, constructed with 7-nm technology. 16 The chip integrates power management with improved model performance and power use. Codesign approaches integrating special AI models and hardware to achieve energy efficiency are also noteworthy.…”

Section: Artificial Intelligence/machine Learningmentioning

confidence: 99%

Colors of Artificial Intelligence

Lin¹

2021

Computer

View full text Add to dashboard Cite

On 22 and 23 April, at the two-day Leaders' Summit on Climate 2021, global heads of state announced ambitious targets to manage the climate crisis. It was announced that, by 2030, the United States would cut its greenhouse gas emissions by 50%. The European Union targeted a 55% reduction in net greenhouse gases by 2030. China promised to strengthen its controls on greenhouse gas emissions beyond carbon dioxide and decelerate coal consumption. Conversely, according to a 2019 estimation, in the field of artificial intelligence (AI), the carbon footprint of training a state-of-the-art language model equates to that of five U.S. cars' entire

show abstract

9.1 A 7nm 4-Core AI Chip with 25.6TFLOPS Hybrid FP8 Training, 102.4TOPS INT4 Inference and Workload-Aware Throttling

Cited by 61 publications

References 6 publications

Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

4-bit Quantization of LSTM-based Speech Recognition Models

Colors of Artificial Intelligence

Contact Info

Product

Resources

About