8-bit Numerical Formats for Deep Neural Networks

Noune, Badreddine; Jones, Peter B.; Justus, Daniel; Masters, Dominic; Luschi, C.

doi:10.48550/arxiv.2206.02915

Cited by 4 publications

(14 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For range, the same techniques used to train in FP16 are required, and for precision, the use of FP8 has thus far been restricted to only the inputs of matmul (matrix multiply) operations (Sun et al, 2019;Noune et al, 2022;Micikevicius et al, 2022), with 3 mantissa bits typically required for weights and activations, and 2 mantissa bits for gradients.…”

Section: Floating-point Formats For Deep Learningmentioning

confidence: 99%

“…By 'training in FP8' we mean that matmuls are performed in FP8 (inputs are cast down to FP8, with outputs in higher precision) with wider formats typically used elsewhere, following the lead of Sun et al (2019); Noune et al (2022) and Micikevicius et al (2022). FP8 reduces both precision and range, and has not generally been used for other operations as matmuls benefit most from using low-precision formats.…”

Section: Low-precision Training Techniquesmentioning

confidence: 99%

“…Below: A histogram of exponent values at initialisation for the above FFN, with shade indicating bin density. The y-axis reflects exponent values available in FP16, while dashed lines show the max/min exponents of the FP8 E4 format of Noune et al (2022).…”

Section: Introductionmentioning

confidence: 99%

“…required (Noune et al, 2022;Micikevicius et al, 2022;Kuzmin et al, 2022). The traditional approach of using 32bit floats is being superseded by mixed precision strategies, which place many values into 16-bit formats (Micikevicius et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

“…The traditional approach of using 32bit floats is being superseded by mixed precision strategies, which place many values into 16-bit formats (Micikevicius et al, 2018). Furthermore, 8-bit floating-point hardware is becoming available (Graphcore, 2022;Nvidia, 2022), with the potential for accurate 8-bit training already demonstrated (Wang et al, 2018;Sun et al, 2019;Noune et al, 2022;Micikevicius et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Unit Scaling: Out-of-the-Box Low-Precision Training

Blake¹,

Orr²,

Luschi³

2023

Preprint

View full text Add to dashboard Cite

We present unit scaling, a paradigm for designing deep learning models that simplifies the use of low-precision number formats. Training in FP16 or the recently proposed FP8 formats offers substantial efficiency gains, but can lack sufficient range for out-of-the-box training. Unit scaling addresses this by introducing a principled approach to model numerics: seeking unit variance of all weights, activations and gradients at initialisation. Unlike alternative methods, this approach neither requires multiple training runs to find a suitable scale nor has significant computational overhead. We demonstrate the efficacy of unit scaling across a range of models and optimisers. We further show that existing models can be adapted to be unit-scaled, training BERT LARGE in FP16 and then FP8 with no degradation in accuracy.

show abstract

Section: Floating-point Formats For Deep Learningmentioning

confidence: 99%

Section: Low-precision Training Techniquesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Unit Scaling: Out-of-the-Box Low-Precision Training

Blake¹,

Orr²,

Luschi³

2023

Preprint

View full text Add to dashboard Cite

show abstract

Climate‐change modelling at reduced floating‐point precision with stochastic rounding

Kimpson

Paxton

Chantry

et al. 2023

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

Reduced‐precision floating‐point arithmetic is now deployed routinely in numerical weather forecasting over short timescales. However, the applicability of these reduced‐precision techniques to longer‐timescale climate simulations—especially those that seek to describe a dynamical, changing climate—remains unclear. We investigate this question by deploying a global atmospheric, coarse‐resolution model known as Simplified Parameterizations PrimitivE Equation DYnamics (SPEEDY) to simulate a changing climate system subject to increased CO2$$ {\mathrm{CO}}_2 $$ concentrations, over a 100‐year timescale. Whilst double precision is typically the operational standard for climate modelling, we find that reduced‐precision solutions are sufficiently accurate. Rounding the floating‐point numbers stochastically, rather than using the more common “round‐to‐nearest” technique, improves the performance of the reduced‐precision solutions notably. Over 100 years, the mean bias error (MBE) in the global mean surface temperature (precipitation) relative to the double‐precision solution is +1.8 prefix×10prefix−2$$ \times 1{0}^{-2} $$ K (prefix−8prefix×10prefix−4$$ -8\times 1{0}^{-4} $$ mm·$$ \cdotp $$(6 hr)prefix−1$$ {}^{-1} $$) when integrating numerically at half precision (10 significant bits) with stochastic rounding. By examining the resultant climatic distributions that arise after 100 years, the difference in the expected value of the global surface temperature relative to the double‐precision solution is ≤5prefix×10prefix−3$$ \le 5\times 1{0}^{-3} $$ K and that for precipitation is 8prefix×10prefix−4$$ 8\times 1{0}^{-4} $$ mm·$$ \cdotp $$(6 hr)prefix−1$$ {}^{-1} $$. Whilst further research is necessary to extended these results to more complex and higher‐resolution models, they indicate that reduced‐precision techniques and stochastic rounding could be suitable for the next generation of climate models and motivate the use of low‐precision hardware to this end.

show abstract

Floating-point arithmetic

Boldo¹,

Jeannerod²,

Melquiond³

et al. 2023

Acta Numerica

View full text Add to dashboard Cite

Floating-point numbers have an intuitive meaning when it comes to physics-based numerical computations, and they have thus become the most common way of approximating real numbers in computers. The IEEE-754 Standard has played a large part in making floating-point arithmetic ubiquitous today, by specifying its semantics in a strict yet useful way as early as 1985. In particular, floating-point operations should be performed as if their results were first computed with an infinite precision and then rounded to the target format. A consequence is that floating-point arithmetic satisfies the ‘standard model’ that is often used for analysing the accuracy of floating-point algorithms. But that is only scraping the surface, and floating-point arithmetic offers much more.In this survey we recall the history of floating-point arithmetic as well as its specification mandated by the IEEE-754 Standard. We also recall what properties it entails and what every programmer should know when designing a floating-point algorithm. We provide various basic blocks that can be implemented with floating-point arithmetic. In particular, one can actually compute the rounding error caused by some floating-point operations, which paves the way to designing more accurate algorithms. More generally, properties of floating-point arithmetic make it possible to extend the accuracy of computations beyond working precision.

show abstract

8-bit Numerical Formats for Deep Neural Networks

Cited by 4 publications

References 30 publications

Unit Scaling: Out-of-the-Box Low-Precision Training

Unit Scaling: Out-of-the-Box Low-Precision Training

Climate‐change modelling at reduced floating‐point precision with stochastic rounding

Floating-point arithmetic

Contact Info

Product

Resources

About