Evaluations on Deep Neural Networks Training Using Posit Number System

Lu, Jinming; Fang, Chao; Xu, Mingyang; Lin, Jun; Wang, Zhongfeng

doi:10.1109/tc.2020.2985971

Cited by 47 publications

(20 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is particularly problematic when the model weights are updated with low-precision posits, since they do not have enough resolution for small numbers. As suggested in [17], using 16-bit posits for the optimizer and loss is usually enough to allow models to train with low-precision posits. With this observation in mind, this model was trained with a different precision for the optimizer and loss, while using posit(8, 2) everywhere else (see Table 4).…”

Section: Positmentioning

confidence: 99%

“…Later, in [14,15], a FCNN was trained for MNIST and Fashion MNIST using {16, 32}-bit posits. In [16,17], Convolutional Neural Networks (CNNs) were trained using a mix of {8, 16}-bit posits, but still relying on floats for the first epoch and layer computations. More recently, in [18], a CNN was trained for CIFAR-10 but using only {16, 32}-bit posits.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Positnn: Training Deep Neural Networks with Mixed Low-Precision Posit

Raposo

Pfister

Roma

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Low-precision formats have proven to be an efficient way to reduce not only the memory footprint but also the hardware resources and power consumption of deep learning computations. Under this premise, the posit numerical format appears to be a highly viable substitute for the IEEE floating-point, but its application to neural networks training still requires further research. Some preliminary results have shown that 8-bit (and even smaller) posits may be used for inference and 16-bit for training, while maintaining the model accuracy. The presented research aims to evaluate the feasibility to train deep convolutional neural networks using posits. For such purpose, a software framework was developed to use simulated posits and quires in end-to-end training and inference. This implementation allows using any bit size, configuration, and even mixed precision, suitable for different precision requirements in various stages. The obtained results suggest that 8bit posits can substitute 32-bit floats during training with no negative impact on the resulting loss and accuracy.

show abstract

Section: Positmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Positnn: Training Deep Neural Networks with Mixed Low-Precision Posit

Raposo

Pfister

Roma

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Another very promising alternative to IEEE 32-bit Floating-point standard is the posit TM number system, proposed by Gustafson [19]. This format has been proven to match single precision accuracy performance with only 16 bits used for the representation [9,12,16,17,24]. Furthermore, the first hardware implementations of this novel type are very promising in terms of energy consumption and area occupation [10,20,28].…”

Section: Introductionmentioning

confidence: 99%

Vectorizing posit operations on RISC-V for faster deep neural networks: experiments and comparison with ARM SVE

Cococcioni

Rossi

Ruffaldi³

et al. 2021

Neural Comput & Applic

View full text Add to dashboard Cite

With the arrival of the open-source RISC-V processor architecture, there is the chance to rethink Deep Neural Networks (DNNs) and information representation and processing. In this work, we will exploit the following ideas: i) reduce the number of bits needed to represent the weights of the DNNs using our recent findings and implementation of the posit number system, ii) exploit RISC-V vectorization as much as possible to speed up the format encoding/decoding, the evaluation of activations functions (using only arithmetic and logic operations, exploiting approximated formulas) and the computation of core DNNs matrix-vector operations. The comparison with the well-established architecture ARM Scalable Vector Extension is natural and challenging due to its closedness and mature nature. The results show how it is possible to vectorize posit operations on RISC-V, gaining a substantial speed-up on all the operations involved. Furthermore, the experimental outcomes highlight how the new architecture can catch up, in terms of performance, with the more mature ARM architecture. Towards this end, the present study is important because it anticipates the results that we expect to achieve when we will have an open RISC-V hardware co-processor capable to operate natively with posits.

show abstract

“…It uses a regime field to further scale the exponent and thus can provide much larger dynamic range than the floating-point formats. This can benefit many applications, such as deep learning training [3], where a large dynamic range is expected from the numeric format. In addition, posit number format uses tapered precision, and thus its number distribution is non-uniformed.…”

Section: Introductionmentioning

confidence: 99%

Efficient Multiple-Precision Posit Multiplier

Zhang

2021

2021 IEEE International Symposium on Circuits and Systems (ISCAS)

View full text Add to dashboard Cite

Posit number system has been recently widely applied in many fields of applications. For different applications, the precision requirements are usually different. In addition, the transprecision computing paradigm, which is proposed for energy efficient computation, even requires different precision in each computation step. To support computations of various precision in a single hardware architecture, in this paper, a unified architecture of multiple-precision posit multiplier is proposed. The proposed posit multiplier supports the commonly used Posit(8, 0), Posit(16, 1), and Posit(32, 2) formats, where one Posit(32, 2), or two parallel Posit(16, 1), or four parallel Posit(8, 0) multiplications can be accomplished each time. Each module of the proposed posit multiplier is carefully tailored for resource sharing among three supported precision formats. Compared to the Posit(32, 2) multiplier, the proposed multipleprecision multiplier adds the support for parallel low-precision posit multiplications with only 12.8% more area and 15.4% more power. The proposed architecture can be used in posit-enabled general-purpose processor designs.

show abstract

Evaluations on Deep Neural Networks Training Using Posit Number System

Cited by 47 publications

References 23 publications

Positnn: Training Deep Neural Networks with Mixed Low-Precision Posit

Positnn: Training Deep Neural Networks with Mixed Low-Precision Posit

Vectorizing posit operations on RISC-V for faster deep neural networks: experiments and comparison with ARM SVE

Efficient Multiple-Precision Posit Multiplier

Contact Info

Product

Resources

About