Optimal visual–haptic integration with articulated tools

Abstract

When we feel and see an object, the nervous system integrates visual and haptic information optimally, exploiting the redundancy in multiple signals to estimate properties more precisely than is possible from either signal alone. We examined whether optimal integration is similarly achieved when using articulated tools. Such tools (tongs, pliers, etc) are a defining characteristic of human hand function, but complicate the classical sensory ‘correspondence problem’ underlying multisensory integration. Optimal integration requires establishing the relationship between signals acquired by different sensors (hand and eye) and, therefore, in fundamentally unrelated units. The system must also determine when signals refer to the same property of the world—seeing and feeling the same thing—and only integrate those that do. This could be achieved by comparing the pattern of current visual and haptic input to known statistics of their normal relationship. Articulated tools disrupt this relationship, however, by altering the geometrical relationship between object properties and hand posture (the haptic signal). We examined whether different tool configurations are taken into account in visual–haptic integration. We indexed integration by measuring the precision of size estimates, and compared our results to optimal predictions from a maximum-likelihood integrator. Integration was near optimal, independent of tool configuration/hand posture, provided that visual and haptic signals referred to the same object in the world. Thus, sensory correspondence was determined correctly (trial-by-trial), taking tool configuration into account. This reveals highly flexible multisensory integration underlying tool use, consistent with the brain constructing internal models of tools’ properties.Electronic supplementary materialThe online version of this article (doi:10.1007/s00221-017-4896-5) contains supplementary material, which is available to authorized users.

Somatosensory-guided tool use modifies arm representation for action

“…In principle, both vision and somatosensation could contribute to tool incorporation, and recent work has highlighted their integration 21 . While vision seems to be predominant for space and objects coding since infancy 22–24 , proprioceptive and tactile signals appear to be critical information for the update of this implicit action-related body representation.…”

Multilayer Haptic Feedback for Pen-Based Tablet Interaction

“…In most cases the visual display serves as reference point for feedback in the flexible layer. As visual and haptic feedback are aligned, feedback will likely be mentally integrated, as predicted by the visual-haptic integration theory [76]. Yet, the flexible haptic layer can also serve as reference point for the visual display, guiding user actions.…”

Haptic Hybrid Prototyping (HHP): An AR Application for Texture Evaluation with Semantic Content in Product Design

“…It is important to point out that this experience is not only linked to the haptic sense, but to the union of this sense with the sense of view. Accordingly, in the mind of the user this sensory information is linked to the product in an inseparable way [47,48] so that it was this texture associated with that image that led him or her to transmit a semantic content and with it, evoke an emotion [49] and not the perception of both pieces of information separately. Thus, the design team of the trading company can use this sensory and cognitive experience to communicate through its products a concrete brand image [50,51] detecting this synergetic effect of two sensory channels together with a concrete memory and an associated emotion.…”