Exploring crossmodal correspondences for future research in human movement augmentation

Abstract

“Crossmodal correspondences” are the consistent mappings between perceptual dimensions or stimuli from different sensory domains, which have been widely observed in the general population and investigated by experimental psychologists in recent years. At the same time, the emerging field of human movement augmentation (i.e., the enhancement of an individual’s motor abilities by means of artificial devices) has been struggling with the question of how to relay supplementary information concerning the state of the artificial device and its interaction with the environment to the user, which may help the latter to control the device more effectively. To date, this challenge has not been explicitly addressed by capitalizing on our emerging knowledge concerning crossmodal correspondences, despite these being tightly related to multisensory integration. In this perspective paper, we introduce some of the latest research findings on the crossmodal correspondences and their potential role in human augmentation. We then consider three ways in which the former might impact the latter, and the feasibility of this process. First, crossmodal correspondences, given the documented effect on attentional processing, might facilitate the integration of device status information (e.g., concerning position) coming from different sensory modalities (e.g., haptic and visual), thus increasing their usefulness for motor control and embodiment. Second, by capitalizing on their widespread and seemingly spontaneous nature, crossmodal correspondences might be exploited to reduce the cognitive burden caused by additional sensory inputs and the time required for the human brain to adapt the representation of the body to the presence of the artificial device. Third, to accomplish the first two points, the benefits of crossmodal correspondences should be maintained even after sensory substitution, a strategy commonly used when implementing supplementary feedback.

Marketing sonified fragrance: Designing soundscapes for scent

“…Despite the recent hype regarding the matching of scent and sound, it remains to be seen whether this innovative new multisensory approach to the marketing and experience of scented products, such as perfume, will last in the long-term, or whether instead it is just another one of those sensory marketing fads that will pass once the headlines and media coverage have receded. However, looking to the future, it would seem possible that crossmodal correspondences between fragrance and sound might one day also be exploited in the context of sensory substitution research ( Pinardi et al, 2023 ). For example, assuming that correspondences between olfactory fragrance notes and acoustic features of music are well-established, one could imagine conveying auditory information using odors, or vice versa , to allow people with impaired hearing/olfaction to experience fragrances/music more fully (i.e., in a manner that more fully engages the senses).…”

Impact of supplementary sensory feedback on the control and embodiment in human movement augmentation

“…After extensive training, optimized internal models could allow participants to rely on inverse models for implementing feedforward controls, reducing the need for sensory feedback 107 . A very informative yet unintuitive sensory cue (e.g., SRL joint torque) might require longer training before it can be learned and produce its benefits 84 , while a supplementary feedback that exploits spontaneous associations might drastically reduce the need for training 142 . Hence, learning time should be a main aspect to consider when comparing the efficacy of different feedbacks.…”