Towards an Audio-Locative Mobile Application for Immersive Storytelling

Abstract

We live in an age in which digital media is omnipresent and augmented reality is beginning to find its way into our everyday lives, GPS allows us to determine our position with meter precision and the sensor capabilities of smartphones are increasing. All these technologies in combination enable us to explore one of the oldest human art forms in a new way: storytelling. The presented work aims at developing approaches that combine different elements of audio media playback, geolocation and other sensor capabilities of smartphones to allow the creation of immersive geolocated narratives within a mobile application. Thus, the narrative can be precisely adjusted to the specific spatial and temporal context of the user. One of the main goals of such a geolocated narrative would be to influence the cognitive processing of the users. As the visual perception of the location stays the same while using the app, it is only the auditory cognition that causes a subtle enhancement and thus alteration of the reality which could result in a very strong sense of immersion. There are many location-based audio applications on the market. Most of them are intended for tourism in the form of an audio tour guide. Some of those applications have been analyzed before developing our own ideas on how to intertwine sound, space and time into an immersive mobile application for storytelling with semi-linear narrative structures including a map. Possible features could be influenceability of the narrative by, e.g., speed or approach direction of the user, sound movement over time on predetermined routes, etc. The first prototype implemented some audiovisual presentations among others. Keywords Audiovisual cartography • Multimedia cartography • Locative media • Storytelling • Mobile mapping Zusammenfassung Wir leben in einem Zeitalter, in welchem digitale Medien omnipräsent sind und Augmented Reality Einzug ins tägliche Leben erhält, in dem GPS uns befähigt, unsere Position mit Meter-Genauigkeit zu bestimmen und sich das Sensor-Potential von Smartphones erhöht. Das Zusammenspiel all dieser Technologien erlaubt es, eine der ältesten Kunstformen neu zu entdecken: die Erzählkunst. Die vorliegende Arbeit hat zum Ziel, Ansätze zu entwickeln, welche verschiedene Elemente der Audio-Wiedergabe, Geolokalisierung und andere Sensor-Fähigkeiten kombinieren, um immersive, räumlich verortete Narrative für eine mobile Anwendung zu kreieren. Damit können Geschichten präzise an den spezifischen räumlich-zeitlichen Kontext des Nutzers angepasst werden. Die Hauptabsicht einer solchen ortbezogenen Geschichte ist es, die kognitive Wahrnehmung des Nutzers zu beeinflussen. Da während der App-Nutzung die visuelle Wahrnehmung der Umgebung gleich bleibt, ist es nur die auditive Wahrnehmung, welche eine subtile Anreicherung und damit eine Veränderung der Realität bewirkt, was ein starkes immersives Erleben hervorrufen kann. Es sind zahlreiche ortbasierte Audio-Anwendungen auf dem Markt. Bei den meisten handelt es sich um touristische Audio-Führer. Einige dieser Anwen...

Concept of the Intelligent Guide with AR Support

“…The study [22] explored the useful features of a virtual hands-free guide, without eyes, which can answer questions using the interface for conversational dialogue and inform the user about interesting features while driving, the tourist in different directions. The work [23] is aimed at developing approaches that combine different elements of sound reproduction, geolocation and other sensory capabilities of smartphones, which allows to create exciting geolocation stories in a mobile application.…”

“…Unlike analogues [12,22,23], the proposed concept of the intelligent guide is based on a combination of the following technologies: identification, AR, GMS, IoT, sound effects support and intelligent voice chatbot.…”

“…It is expected, the modified version of Intelligent Guide will be integrated with the AURA project MagneticOne [23]. AURA will be a WEB platform that allows to buy tickets for VR viewing performances for opera houses, which are currently represented by 3D model for theaters in Florence, Berlin and Lviv.…”

Raumbezogene Visualisierung trifft auf Landschaftsforschung und „Pinballology“: Beispiele der Landschaftskonstruktion in Flipperspielen

“…Thanks to developments in neighboring academic disciplines and related industries, such as gaming, informatics and (other) engineering fields, the opportunities for enhancing the visualization of geographic space are undergoing a permanently ongoing development. This leads to new variants of media that can include many new facets and features, such as high-quality sound (e.g., Berger and Bill 2019;Edler and Vetter 2019;Indans et al 2019;Hruby 2019;Lammert-Siepmann et al 2017;Schito and Fabrikant 2018), information graphics (Dunlap and Lowenthal 2016;Meier 2017;Schiewe 2017), 3D animation (Edler et al 2018a;Kersten et al 2018;Lovett et al 2015), and gamified interaction techniques (Ahlqvist 2011;Kapenekakis and Chorianopoulos 2017;Lütjens et al 2019)-to name just a few examples that are intensively discussed in recent international literature of cartography and related disciplines.…”

Immersive VR Experience of Redeveloped Post-industrial Sites: The Example of “Zeche Holland” in Bochum-Wattenscheid

“…In addition to the visual animation part, a sound sequence is played. This auditory element has a clear function to communicate specific semantical information, whereas (3D) sound in this VR application is otherwise used to create an audiorealistic impression of the soundscape based on atmospheric field recordings (see also Berger and Bill 2019;Indans et al 2019;Laakso and Sarjakoski 2010;Schafer 1977). The sound animation at this location includes a vocal message (see Sherman and Craig 2019, p. 438).…”