SegTest: Metamorphic Testing of Image Segmentation via Guided Instance‐Level Test Data Augmentation

Abstract

Image segmentation software (SegSoftware) is a kind of DNN‐based image analysis software that aims to recognize the shapes and categories of instances according to their implicit semantic information. SegSoftware frequently uses in safety‐critical fields. Therefore, we should provide adequate testing to SegSoftware. Due to the high cost of manually acquiring the testing oracle for SegSoftware, we employ metamorphic testing to detect its erroneous behaviour. This paper proposes SegTest, a metamorphic testing method that primarily addresses two major challenges in applying metamorphic testing to SegSoftware: (1) devising a method for generating derived test cases, which is the data augmentation approach, and (2) finding effective metamorphic relations for automatically generating the testing oracle. Regarding the former, SegTest utilizes an instance‐level data augmentation method. It generates new test data by inserting annotated instances into the existing images. For ease of exposing erroneousness, we statistically analysed thousands of SegSoftware erroneous behaviours and formulated the guidance strategy of instance selecting and insertion positioning. As for the latter, this paper proposes a metamorphic relation to insert an instance at a position in an original image, where SegSoftware should accurately segment the inserted instance's contour and assign it the appropriate category while preserving the segmentation results of other regions unchanged. Our empirical study shows that SegTest can effectively detect thousands of erroneous behaviours of SegSoftware, and the formulated augmentation strategy achieves a 12.1%–14.1% improvement in SegSoftware erroneousness detection. SegTest also detects 7135 erroneous behaviours on the commercial IBM Segmenter, which verifies the effectiveness of erroneousness detection in practice.