Estimation error in image quality measurements

Abstract

The development and adoption of standard image quality measurement and analysis methods have helped both the evaluation of competing imaging products and in technologies. Inherent in the interpretation of results from any particular evaluation, however, are the variation of the method itself, the sampling of test images, equipment, and test conditions. Here we take a statistical approach to measurement variation, and interpret the objective as being the estimation of particular system or image properties, based on data, collected as part of standard testing. Measurement variation was investigated for two signal-transfer methods commonly used for digital camera and scanner evaluation: the ISO 12233 slanted-edge spatial frequency response and the dead-leaves method for texture-MTF evaluation being developed by the Camera Phone Image Quality (CPIQ) Initiative. In each case, the variation due to the selection of analysis regions was computed by repeated analysis. The slanted-edge methods indicated a relative error in the range of 1-3% depending on the nature of the region selection. For the dead-leaves method, the amplitude spectrum (square root of the noise-power spectrum) showed a relative error of approximately 4-6%, however, this can be reduced by applying spectral estimation methods commonly used in image noise analysis.

Validation of Modulation Transfer Functions and Noise Power Spectra from Natural Scenes

“…We suggest that further analysis of such error is needed to evaluate the "correctness" of each SPD-MTF and SPD-NPS measure. In the case of the SPD-MTFs, this may involve adapting error propagation methods for existing dead leaves MTF measurement implementations [50].…”

Texture MTF from images of natural scenes

“…Unlike the Canon system, where pictorial test images produced MTFs that differ only at high frequencies, MTF discrepancies here can be noticed at low frequencies as well. These discrepancies exceed the 4-6% relative error, characteristic of the dead leaves method [27]. The grass test image produced a curve showing a better system performance overall than the dead leaves, while the tree a better performance at low frequencies but poorer at high frequencies (Fig.…”

Camera Resolution and Distortion: Advanced Edge Fitting

“…Given this measurement variation, and the previously discussed bias due to image distortion leads us to a statistical approach. 6 We consider the fitting of the edge location, whether to a first or higher-order function as an estimation problem. As for any statistical modelling effort, examining the remaining residual error is useful.…”