"I Always Wanted to See the Night Sky"

Abstract

Sensory Substitution Devices (SSDs) convert visual information into another sensory channel (e.g. sound) to improve the everyday functioning of blind and visually impaired persons (BVIP). However, the range of possible functions and options for translating vision into sound is largely open-ended. To provide constraints on the design of this technology, we interviewed ten BVIPs who were briefly trained in the use of three novel devices that, collectively, showcase a large range of design permutations. The SSDs include the 'Depth-vOICe,' 'Synaestheatre' and 'Creole' that offer high spatial, temporal, and colour resolutions respectively via a variety of sound outputs (electronic tones, instruments, vocals). The participants identified a range of practical concerns in relation to the devices (e.g. curb detection, recognition, mental effort) but also highlighted experiential aspects. This included both curiosity about the visual world (e.g. understanding shades of colour, the shape of cars, seeing the night sky) and the desire for the substituting sound to be responsive to movement of the device and aesthetically engaging.

Towards Multisensory Storytelling with Taste and Flavor

“…vision) into another sense (e.g. hearing or touch, see [5,24]). They are typically developed for blind or deaf people, but what if we take the concept of sensory substitution further to introduce a new way to emotionally engage with content, instead of seeing you taste a scene with sound, and flavors that become part of the narrative and unfold over time.…”

“…We reflect on the natural marriage that has developed between certain foods and film experiences. Based on this reflection, we take inspiration from existing research on topics such as, sensory substitution ('seeing through sound', e.g., [24]), to suggest a new approach to multisensory film. We aim to position this manuscript as a first point of contact for those interested in multisensory film experiences, in particular, those involving taste, flavor, and food.…”

How Much Spatial Information Is Lost in the Sensory Substitution Process? Comparing Visual, Tactile, and Auditory Approaches

“…Overall, the experience of vision continues to influence our representation of space even if no visual input is currently available. As such, providing structured spatial information to the fully blind not only serves as an intervention to assist in daily tasks (Elli, Benetti, & Collignon, 2014; Hamilton-Fletcher, Obrist, Watten, Mengucci, & Ward, 2016; Kristjánsson et al., 2016; Maidenbaum, Abboud, & Amedi, 2014), but it allows users to interpret their remaining senses more accurately (Cappagli, Finocchietti, Baud-Bovy, Cocchi, & Gori, 2017) and explore how our metamodal representation of space can develop through additional sensory stimulation (Pasqualotto & Esenkaya, 2016).…”

“…Over time, these can even evoke a qualitative shift in user perception so that sound or touch can allow the user to perceive spatialised patterns of light (Ortiz et al., 2011; Ward & Meijer, 2010), a form of synthetically acquired synaesthesia comparable to vision (Ward & Wright, 2014). SSDs not only have the potential to provide access to new spatial information and shift metamodal representations of space, but interviews with users suggest a range of practical benefits outside of the lab including enhanced safety, independence, and facilitating curiosity of the environment, further encouraging spatial exploration (Hamilton-Fletcher, Obrist, et al, 2016). Visuospatial SSDs tend to fall under two main types of feedback, haptic (the active use of touch), or auditory (the sense of listening), and while we explore specific examples of these SSDs later, wider overviews of SSDs are available (Gori et al., 2016; Hamilton-Fletcher & Ward, 2013).…”

“…When objects within the field of view (FoV) of a 3D sensor are sonified in a manner that replicates the perception of sounds emanating from the location of the object, this process is also referred to as a “virtual acoustic space” (Eckert, Blex, & Friedrich, 2018; González-Mora, Rodriguez-Hernandez, Burunat, Martin, & Castellano, 2006; González-Mora, Rodriguez-Hernandez, Rodriguez-Ramos, Díaz-Saco, & Sosa, 1999; Rodríguez-Hernández et al., 2010). One modern incarnation of this is the “Synaestheatre” that converts a depth image from a 3D sensor into realistically spatialised sounds (Hamilton-Fletcher, Obrist, et al, 2016). The sounds are spatialised using head-related transfer function (HRTF), which describes how the spatial positions of the listener and sound source alter the received sounds in terms of interaural timing, intensity, as well as distortions created by the head and pinnae (Algazi, Avendano, & Duda, 2001; Kistler & Wightman, 1992; Kulkarni, Isabelle, & Colburn, 1999; Potisk, 2015).…”