Solving visual object ambiguities when pointing: an unsupervised learning approach

Abstract

Whenever we are addressing a specific object or refer to a certain spatial location, we are using referential or deictic gestures usually accompanied by some verbal description. Particularly, pointing gestures are necessary to dissolve ambiguities in a scene and they are of crucial importance when verbal communication may fail due to environmental conditions or when two persons simply do not speak the same language. With the currently increasing advances of humanoid robots and their future integration in domestic domains, the development of gesture interfaces complementing human–robot interaction scenarios is of substantial interest. The implementation of an intuitive gesture scenario is still challenging because both the pointing intention and the corresponding object have to be correctly recognized in real time. The demand increases when considering pointing gestures in a cluttered environment, as is the case in households. Also, humans perform pointing in many different ways and those variations have to be captured. Research in this field often proposes a set of geometrical computations which do not scale well with the number of gestures and objects and use specific markers or a predefined set of pointing directions. In this paper, we propose an unsupervised learning approach to model the distribution of pointing gestures using a growing-when-required (GWR) network. We introduce an interaction scenario with a humanoid robot and define the so-called ambiguity classes. Our implementation for the hand and object detection is independent of any markers or skeleton models; thus, it can be easily reproduced. Our evaluation comparing a baseline computer vision approach with our GWR model shows that the pointing-object association is well learned even in cases of ambiguities resulting from close object proximity.

Robust Multi-User In-Hand Object Recognition in Human-Robot Collaboration Using a Wearable Force-Myography Device

“…Similarly, EMG and IMU were used on the user's forearm to control a drone through obstacles [38]. However, IMU is prone to drift and requires frequent calibrations [17]. A different approach used…”

“…As discussed above, prior pointing work considered the measurement of either the forearm (FA) vector formed by the wrist and elbow points [16], [28], index finger (IF) direction [17] or the direction of the vector connecting the user eyes (root of nose) and finger (EF) [16], [26], [33]. Illustration of these measurement types is seen in Figure 3.…”

“…While depth data enables easier access to pointing gestures, depth cameras are limited to short-range in indoor environments and work poorly outdoors. In addition, they require an additional hardware setup and limit the generality of the method [17]. However, only few addressed the pointing problem using a single RGB camera [18].…”

“…Features from both the arm and head were extracted and initiation of estimation was triggered using voice commands. In [17], detection of whether the hand is pointing on objects was performed. The detection was conducted with a camera set above the table in a constructed environment.…”