Integration of vision and haptics during tool use

Abstract

When integrating signals from vision and haptics the brain must solve a "correspondence problem" so that it only combines information referring to the same object. An invariant spatial rule could be used when grasping with the hand: here the two signals should only be integrated when the estimate of hand and object position coincide. Tools complicate this relationship, however, because visual information about the object, and the location of the hand, are separated spatially. We show that when a simple tool is used to estimate size, the brain integrates visual and haptic information in a near-optimal fashion, even with a large spatial offset between the signals. Moreover, we show that an offset between the tool-tip and the object results in similar reductions in cross-modal integration as when the felt and seen positions of an object are offset in normal grasping. This suggests that during tool use the haptic signal is treated as coming from the tool-tip, not the hand. The brain therefore appears to combine visual and haptic information, not based on the spatial proximity of sensory stimuli, but based on the proximity of the distal causes of stimuli, taking into account the dynamics and geometry of tools.

Stroking me softly: Body-related effects in effect-based action control

“…The present apparatus does, indeed, share several features with mechanical tools, with the incompatible R-E relation being similar to a firstclass lever with one pivot, in which hand movements in a certain direction (say, left) are continuously translated into brush movements to the opposite side (say, right). The difficulty associated with the R-E incompatible condition could thus be seen as a difficulty that arises from representing a directional tool transformation (e.g., Beisert, Massen, & Prinz 2010;Müsseler & Skottke, 2011;Takahashi, Diedrichsen, & Watt, 2009; for reviews of tool use, see Heuer & Sülzenbrück, 2013;Johnson-Frey, 2003).…”

Grab an object with a tool and change your body: tool-use-dependent changes of body representation for action

“…That is, when planning an action, it is the prediction of the outcome rather than speciWc movement features that plays a crucial role (see Ladwig et al 2012 for similar results). As one could expect the same eVect to be present when performing the tracking movement with the hand alone, it could be inferred, as Rieger et al (2005) advanced, that this capacity of the CNS to learn and Xexibly map movements and their consequences plays an important role in tool-use (Takahashi et al 2009). Here, we focus a complementary aspect that we think is crucial for skilful tool-use, that is, the incorporation of the tool into the body representation used to plan and execute actions, that is, the Body Schema.…”

Vision and proprioception in action monitoring by young and older adults

“…Furthermore, the vertical plane of cursor motion and the horizontal plane of hand movement were clearly distinct, with no direct link between them. Both the spatial separation and the absence of a visible link reduce bimodal integration even with object identity (Gepshtein et al, 2005;Takahashi et al, 2009), and they should facilitate bimodal discrimination in tool use where it is a prerequisite of the conscious experience of the inputeoutput relation of the tool.…”