Hallucination In Object Detection — A Study In Visual Part VERIFICATION

Abstract

We show that object detectors can hallucinate and detect missing objects; potentially even accurately localized at their expected, but non-existing, position. This is particularly problematic for applications that rely on visual part verification: detecting if an object part is present or absent. We show how popular object detectors hallucinate objects in a visual part verification task and introduce the first visual part verification dataset: DelftBikes 1 , which has 10,000 bike photographs, with 22 densely annotated parts per image, where some parts may be missing. We explicitly annotated an extra object state label for each part to reflect if a part is missing or intact. We propose to evaluate visual part verification by relying on recall and compare popular object detectors on DelftBikes.

Humans Disagree With the IoU for Measuring Object Detector Localization Error

“…We address two important features of object localization: (i) Box Size and (ii) Box Position, which are affected by the IoU score, in four online user studies (two studies per feature). 2 We also experiment with various object sizes (small -S, medium -M, large -L) 3 and IoU values (0.3, 0.5, 0.7, 0.9) to study differences and similarities between humans and detection algorithms.…”

“…Object location can be used as a first step for a downstream task, e.g., instance segmentation [1], or human pose estimation [2]. Alternatively, in this paper, we focus on the setting where an object detection is presented to humans as an end result, where examples include visual inspection [3], or focusing attention in medical images [4]. We ⋆ Authors with equal contribution.…”

“…1 (2) We investigate the relationship between a too small bounding box and a too large bounding box, where they both have the same IoU score. (3) We analyze the impact of object symmetry and bounding box position in human preference and acceptance of detectors' output. (4) We experiment with various object sizes (small, medium, large) and recommend future studies.…”

“…Ethical approval was not required -we do not collect personal identifiers 3. We adopt the definition of object size provided with the MS COCO dataset (https://cocodataset.org/#detection-eval) 4.…”