Improved precision of fixed-point algorithms by means of common factors

Abstract

We describe a general techniques for improving precision of fixed-point implementations of signal processing algorithms (such as filters, transforms, etc.) by introducing "common factors". These factors are applied to groups of real constants that need to be approximated by dyadic rational numbers, and we show that by carefully choosing values of common factors, the errors of final dyadic rational approximations can be significantly reduced. We show that the problem of design of such approximations is related to the classic Diophantine approximation problem, and include examples explaining how it can be solved, and used for improving practical designs.

Continued fractions, diophantine approximations, and design of color transforms

“…† This idea is similar to one that we have previously suggested for implementations of Discrete Cosine Transforms 4,5. …”

Efficient large size transforms for high-performance video coding

“…ξ ς η Such flexibility was already exploited to gain extra precision in the design of fixed-point algorithms in references 5,[18][19][20] .…”