Supplemental subjective testing to evaluate the performance of image and video quality estimators

Abstract

The subjective tests used to evaluate image and video quality estimators (QEs) are expensive and time consuming. More problematic, the majority of subjective testing is not designed to find systematic weaknesses in the evaluated QEs. As a result, a motivated attacker can take advantage of these systematic weaknesses to gain unfair monetary advantage. In this paper, we draw on some lessons of software testing to propose additional testing procedures that target a specific QE under test. These procedures supplement, but do not replace, the traditional subjective testing procedures that are currently used. The goal is to motivate the design of objective QEs which are better able to accurately characterize human quality assessment.

A strategy to jointly test image quality estimators subjectively

“…The strategies in [10,11] can only identify potential false ties. Various strategies in [6] create either potential FTs or potential FOs. The goal of this paper is to create as many False Orderings as possible.…”

“…The boundaries of the regions W o , E o and B o are easy to identify using the QE we are testing; it is straightforward to select an image that has a potential FT [10,11,6]. However, a FT does not provide much information about the severity of a systematic weakness; it only informs us that one exists.…”

“…In [6], we introduce the notion of using one more accurate QE as a proxy for subjective quality when designing test pairs for a second less accurate QE. Occasionally the proxy QE identifies its own weakness.…”

“…The following principles lie at the core of software testing [5] and are directly applicable to testing image and video QEs [6]. The goal of testing should be to find errors, not demonstrate that the system satisfies its specifications.…”

“…In [6], we apply these principles to systematically create targeted, small-scale subjective tests. The test cases are com-prised of image pairs designed to have a high probability of creating potential misclassification errors.…”