Exploiting spatial descriptions in visual scene analysis

Abstract

The reliable automatic visual recognition of indoor scenes with complex object constellations using only sensor data is a nontrivial problem. In order to improve the construction of an accurate semantic 3D model of an indoor scene, we exploit human-produced verbal descriptions of the relative location of pairs of objects. This requires the ability to deal with different spatial reference frames (RF) that humans use interchangeably. In German, both the intrinsic and relative RF are used frequently, which often leads to ambiguities in referential communication. We assume that there are certain regularities that help in specific contexts. In a first experiment, we investigated how speakers of German describe spatial relationships between different pieces of furniture. This gave us important information about the distribution of the RFs used for furniture-predicate combinations, and by implication also about the preferred spatial predicate. The results of this experiment are compiled into a computational model that extracts partial orderings of spatial arrangements between furniture items from verbal descriptions. In the implemented system, the visual scene is initially scanned by a 3D camera system. From the 3D point cloud, we extract point clusters that suggest the presence of certain furniture objects. We then integrate the partial orderings extracted from the verbal utterances incrementally and cumulatively with the estimated probabilities about the identity and location of objects in the scene, and also estimate the probable orientation of the objects. This allows the system to significantly improve both the accuracy and richness of its visual scene representation.

Reference frame selection in dialog: priming or preference?

“…This disagreement and the potential for ambiguities has led to an extensive body of psycholinguistic investigations of which factors contribute to the selection and processing of spatial FOR, mostly using monolog studies. The factors identified range from functional relations between objects (Carlson-Radvansky and Radvansky, 1996) to motion characteristics (Levelt, 1984), gravity (Friederici and Levelt, 1990), priming effects (Watson et al, 2004; Carlson and Van Deman, 2008; Johannsen and de Ruiter, 2013), scene type (Johannsen and de Ruiter, 2013), and properties of the object configuration such as the rotation of the reference object and the position of the located object (Ziegler et al, 2012). …”

“…However, following the attempts to specify FOR preferences (as described above), it has been shown that there is a general preference for the intrinsic FOR in specific object configurations (Ziegler et al, 2012). This study also demonstrated an effect of the located object’s position with regard to the reference object’s FOR.…”