Multiplierless MP-Kernel Machine for Energy-Efficient Edge Devices

Nair, Abhishek Ramdas; Nath, Pallab Kumar; Chakrabartty, Shantanu; Thakur, Chetan Singh

doi:10.1109/tvlsi.2022.3189780

Cited by 8 publications

(1 citation statement)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our experiments, we demonstrated that the area used by the multiplier can be kept small if exceptional value support is modified and the number of mantissa bits is kept small. Still, it may be worthwhile to investigate other changes to training algorithms and number systems, such as multiplierless training where logarithmic numbers are used [46], [51].…”

Section: Discussionmentioning

confidence: 99%

Mixing Low-Precision Formats in Multiply-Accumulate Units for DNN Training

Tatsumi

Filip

White

et al. 2022

2022 International Conference on Field-Programmable Technology (ICFPT)

View full text Add to dashboard Cite

The most compute-intensive stage of deep neural network (DNN) training is matrix multiplication where the multiply-accumulate (MAC) operator is key. To reduce training costs, we consider using low-precision arithmetic for MAC operations. While low-precision training has been investigated in prior work, the focus has been on reducing the number of bits in weights or activations without compromising accuracy. In contrast, the focus in this paper is on implementation details beyond weight or activation width that affect area and accuracy. In particular, we investigate the impact of fixed-versus floating-point representations, multiplier rounding, and floatingpoint exceptional value support. Results suggest that (1) lowprecision floating-point is more area-effective than fixed-point for multiplication, (2) standard IEEE-754 rules for subnormals, NaNs, and intermediate rounding serve little to no value in terms of accuracy but contribute significantly to area, (3) lowprecision MACs require an adaptive loss-scaling step during training to compensate for limited representation range, and (4) fixed-point is more area-effective for accumulation, but the cost of format conversion and downstream logic can swamp the savings. Finally, we note that future work should investigate accumulation structures beyond the MAC level to achieve further gains. • a new, extensible, open-source DNN training framework called Archimedes-MPO which uses FPGA or GPU

show abstract

Section: Discussionmentioning

confidence: 99%