Objective evaluation of segmentation quality using spatio-temporal context

Abstract

In this paper, we propose an automatic method for the objective evaluation of segmentation results. The method is based on computing the deviation of the segmentation results from a reference segmentation. The discrepancy between two results is weighted based on spatial and temporal contextual information, by taking into account the way humans perceive visual information. The metric is useful for applications where the final judge of the quality is a human observer or the results of segmentation are otherwise processed in a human-like fashion. The proposed evaluation has been applied both to automatically provide a ranking among different segmentation algorithms and to optimally set the parameters of a given algorithm.

On Simulating Subjective Evaluation Using Combined Objective Metrics for Validation of 3D Tumor Segmentation

“…The combined objective metrics for segmentation evaluation in [6,7,8,9] were determined empirically. In comparison, the weighing parameters of the new composite metric were computed by quantitatively correlating the objective and subjective measures.…”

“…Compared to the subjective evaluation, the objective metrics are easy to compute, but may not provide an overall assessment of the segmentation quality because each metric only captures certain aspect of the difference between the segmentation and the ground truth. Recently, several methods based on combined objective metrics have been proposed to provide evaluation more relevant to subjective assessment of generic object segmentation [6,7,8,9]. The weights in these combined measures, however, are determined rather empirically, and may not be applicable for tumor segmentation evaluation due to the medical expertise required in the assessment.…”

On Evaluating Video Object Segmentation Quality: A Perceptually Driven Objective Metric

“…In this section, the advantages and disadvantages of state-of-the-art objective methods for segmentation evaluation [15], [20], [22], [20], [23], [24], [22], [28] are presented, none of which includes the characterization of artifact perception in their models. To evaluate a segmented video by discrepancy methods, Erdem and Sankur [25] combined three empirical discrepancy measures into an overall quality segmentation evaluation: misclassification penalty, shape penalty, and motion penalty.…”

“…In this method, the evaluation of the spatial accuracy and the temporal coherence is based on the mean and standard deviation of the 2-D shape estimation errors. In preliminary work, we proposed to evaluate the quality of a segmented object through spatial and temporal accuracy joined to yield a combined metric [15]. This work was based on two other discrepancy methods [23], [27] described below, but did not include any perceptual factor.…”

“…This subjective experiment indicated that both the kind and the size of the hole should be jointly taken into account and not only the distance when an objective metric is proposed. In the objective metrics proposed in the literature, holes are only considered in terms of uncorrelated sets of pixels and of their distances from the reference boundary of the object [15], [22]. With this experiment, it was proved that a cluster of error pixels should be distinguished and their characteristics should be thoroughly studied instead of considering each error pixel individually.…”