3D Audio + Augmented Reality + AI Chatbots + IoT: An Immersive Conversational Cultural Guide

The Role of Audio in Immersive Storytelling: a Systematic Review in Cultural Heritage

“…Commercial headphone equalizers based on HRTFs (Unity or Unreal Engine with Stea-mAudio or FMOD) are used in various works [46, 104, 116, 133, 144-146, 153, 164, 165, 175, 184-186]. Other headphones, commercial or open-source headphone-based rendering models, and commercial or open-source libraries have been considered in reviewed papers: OpenAL [108,109,142], Audiokinetic WWise [155], Apple XCode [175], Processing [130] or not specified [56,110,131,143,150,159,174,187]. Andolina et al [134] uses a "headset capable of 3D audio".…”

“…The most common tracking method (55 papers) selects and personalises audio and video content based on head orientation and position in space, especially in indoor situations. In all outdoor experiences, GPS is used along with compass sensors [55,56,79,108,109,122,135,136,138], otherwise tracking is left unspecified [77,134,138]. GPS is also used in one interactive artwork described "called Hybrid Gifts" [161] in which a smartphone app permits museum visitors to communicate by geolocated audio messages with peers in order to express and transmit their emotions while looking at paintings in a museum.…”

“…Information about the discarded storyline can be obtained only by asking and discussing it with other users. Moreover, artificial intelligence (AI) was employed [159] to adapt storytelling in a VR video game story and, in Exhibot [108,109], to generate storylines about the history of the central square in Heraklion(Greece) based on user position and orientation and third party content services. Surprisingly, the previously cited works were the only contributions using AI techniques, although a large number of works can be found in other interactive and non-linear storytelling contexts.…”

“…Soundscape User model detected by user position and orientation in space [121] User position and/or head orientation [56] Interaction with the virtual environment and AI techniques [165] Virtual Environment and Story User interaction with tangible interface [86,158,160] Interaction with voice [86,160] Directions User position and/or head orientation [79,128,132,135,136] Storyline Interaction with HMD headset controller [108,109,116,128,159,164,185] AI content generation [108,109] Interaction with a tangible device [61] Content created by users User interaction with tangible interface [152] Collaborative experience [161,171] especially in tourist or educational experiences, sound effects are used to create soundscapes and personalized content. In Ec(h)o [61], audio soundscape and content are based on user preferences selected by choosing the orientation of a physical cube.…”

Application of New Technologies in the Promotion and Preservation of Cultural Heritage

“…There's a growing interest in using Audio AR (AAR) at heritage locations, allowing visitors to hear location-specific sounds, akin to 'speaking' historical objects. Simultaneously, cultural institutions are adopting AI chatbots in their audience engagement strategies, facilitating meaningful conversations by answering various user queries (Tsepapadakis & Gavalas, 2023).…”