Processing and compression of noise-dominated data: application to the cosmic microwave background data on board the Planck satellite

Abstract

We address the problem of encoding and compressing data dominated by noise. Information is decomposed into ‘reference’ sequences plus arrays containing noisy differences susceptible to being described by a known probability distribution. One can then give reliable estimates of the optimal compression rates by estimating the corresponding Shannon entropy. As a working example, this idea is applied to an idealized model of the cosmic microwave background (CMB) data on board the Planck satellite. Data reduction is a critical issue in space missions because the total information that can be downloaded to Earth is sometimes limited by telemetry allocation. Similar limitations might arise in remotely operated ground based telescopes. This download‐rate limitation could reduce the amount of diagnostics sent on the stability of the instruments and, as a consequence, curb the final sensitivity of the scientific signal. Our proposal for Planck consists of taking differences of consecutive circles at a given sky pointing. To a good approximation, these differences could be made independent of the external signal, so that they are dominated by thermal (white) instrumental noise, which is simpler to model than the sky signal. Similar approaches can be found in other individual applications. Generic simulations and analytical predictions show that high compression rates, can be obtained with minor or zero loss of sensitivity. Possible effects of digital distortion are also analysed. The proposed scheme is flexible and reliable enough to be optimized in relation to other critical aspects of the corresponding application. For Planck, this study constitutes an important step towards a more realistic modelling of the final sensitivity of the CMB temperature anisotropy maps.

The effect of signal digitisation in CMB experiments

“…Resuming what discussed in Romeo et al (1999), Maris et al (2000a), Gaztañaga et al (2000), and Gaztañaga et al (2001) and using the formalism introduced in Maris et al (2000a), the maximum compression rate, C r , achievable by any lossless compression method for any digitized signal represented by integers of N bits bits and with Shannon's entropy H is (Nelson & Gailly 1996) …”

“…As amply discussed in Romeo et al (1999), Maris et al (2000a), Gaztañaga et al (2000) and Gaztañaga et al (2001) in the case of CMB data, the output of the acquisition chain is white-noise dominated. Taking as a representative case P/LFI, the best compression rate, C r , achievable by compressing the output of the acquisition chain, even in the case of an ideal compressor, is C r < 2.7, to be compared with a required C r > ∼ 8.…”

What Bandwidth Do I Need for My Image?

“…techniques (such as gzip) than do the floating-point originals (Gaztañaga et al 2001;Watson 2002;White & Greenfield 1999;Pence et al 2009;Bernstein et al 2009). In the Δ ¼ 0:5σ representation, after lossless compression, storage and transmission of the image "costs" only a few bits per noisedominated pixel.…”

“…What we call here the "minimum representable difference" has also been called by other authors the "discretization" (Gaztañaga et al 2001) …”

“…This question has been asked before, using information theory, in the context of telemetry (Gaztañaga et al 2001) or image compression (Watson 2002), treating the pixels (or linear combinations of them) as independent. Here we ask this question, in some sense, experimentally, and for the properties of imaging on which optical astronomy depends, where groups of contiguous pixels are used in concert to detect and centroid faint sources.…”