Multisensory Integration as per Technological Advances: A Review

Abstract

Multisensory integration research has allowed us to better understand how humans integrate sensory information to produce a unitary experience of the external world. However, this field is often challenged by the limited ability to deliver and control sensory stimuli, especially when going beyond audio–visual events and outside laboratory settings. In this review, we examine the scope and challenges of new technology in the study of multisensory integration in a world that is increasingly characterized as a fusion of physical and digital/virtual events. We discuss multisensory integration research through the lens of novel multisensory technologies and, thus, bring research in human–computer interaction, experimental psychology, and neuroscience closer together. Today, for instance, displays have become volumetric so that visual content is no longer limited to 2D screens, new haptic devices enable tactile stimulation without physical contact, olfactory interfaces provide users with smells precisely synchronized with events in virtual environments, and novel gustatory interfaces enable taste perception through levitating stimuli. These technological advances offer new ways to control and deliver sensory stimulation for multisensory integration research beyond traditional laboratory settings and open up new experimentations in naturally occurring events in everyday life experiences. Our review then summarizes these multisensory technologies and discusses initial insights to introduce a bridge between the disciplines in order to advance the study of multisensory integration.

Impossible (Food) Experiences in Extended Reality

“…In conclusion, impossible experiences in XR have already demonstrated their value in a wide spectrum of applications: to probe multisensory integration (e.g., Cornelio et al, 2021) and decision making (e.g., Ammann et al, 2020), optimize product/ experience development (e.g., Obrist et al, 2019), and create "fun" consumer experiences (e.g., Barbosa Escobar et al, 2021; https://lepetitchef.com/; see also Wang et al, 2021, for a review).…”

Consumer Consciousness in Multisensory Extended Reality

“…For example, Petit et al (2021) highlighted how AR can facilitate the purchase intention for served (as compared to packaged) food by stimulating mental simulation of the eating process. These effects of visual-enabling technologies can now be reinforced by SETs ( Velasco et al, 2018 ; Petit et al, 2019 ; Velasco and Obrist, 2020 ; Cornelio et al, 2021 ). For example, Heller et al (2019a) reported that including touch control in an AR experience increased the consumer’s willingness to pay.…”

A Reflection on the State of Multisensory Human–Food Interaction Research

“…Indeed, we interface with food through most, if not all our senses (Prescott, 2015;Spence, 2017). Second, technology is ubiquitous and there are growing efforts toward developing multisensory technologies, that is, technologies that are designed to stimulate the human senses beyond audition and vision allow researchers and practitioners to precisely control sensory quality, quantity, and delivery (e.g., haptic stimulation in mid-air, digitally-controlled smell delivery, electric taste devices; Cornelio et al, 2021;Covaci et al, 2018;Obrist et al, 2017;Velasco and Obrist, 2020). This context paved the way for MHFI as an area of inquiry.…”

“…An increasing number of digital technologies are being developed to stimulate our chemical senses and thus create new MHFIs and/ or to study multisensory processes (Cornelio et al, 2021). Indeed, there is increasing interest in the way in which technologies can be used to create and/or augment eating and drinking experiences digitally (Spence et al, 2017;Vi et al, 2017).…”