SoundSight: a mobile sensory substitution device that sonifies colour, distance, and temperature

Abstract

Depth, colour, and thermal images contain practical and actionable information for the blind. Conveying this information through alternative modalities such as audition creates new interaction possibilities for users as well as opportunities to study neuroplasticity. The ‘SoundSight’ App (www.SoundSight.co.uk) is a smartphone platform that allows 3D position, colour, and thermal information to directly control thousands of high-quality sounds in real-time to create completely unique and responsive soundscapes for the user. Users can select the specific sensor input and style of auditory output, which can be based on anything—tones, rainfall, speech, instruments, or even full musical tracks. Appropriate default settings for image-sonification are given by designers, but users still have a fine degree of control over the timing and selection of these sounds. Through utilising smartphone technology with a novel approach to sonification, the SoundSight App provides a cheap, widely accessible, scalable, and flexible sensory tool. In this paper we discuss common problems encountered with assistive sensory tools reaching long-term adoption, how our device seeks to address these problems, its theoretical background, its technical implementation, and finally we showcase both initial user experiences and a range of use case scenarios for scientists, artists, and the blind community.

The Unfolding Space Glove: A Wearable Spatio-Visual to Haptic Sensory Substitution Device for Blind People

“…In addition, there is work that actually did propose SSDs that use depth data, yet they often had problems implementing or testing the systems in practice because the technology was not yet advanced enough (too slow, heavy and/or expensive) [91][92][93][94][95][96][97]. In recent years, however, papers have been published (most of them using sound as an output) that actually proposed and tested applied 3D systems [97][98][99][100][101][102][103][104][105][106][107]. This includes the Sound of Vision device, which today probably ranks among the most advanced and sophisticated, which has also been evaluated in several studies.…”

Accuracy and Usability of Smartphone-Based Distance Estimation Approaches for Visual Assistive Technology Development

“…When PBLV are interviewed about these tools, the most highly sought-after sensory information is distance for objects or people [6] . Recent iPhone Apps like ‘LiDAR Sense’, ‘Super Lidar’, Apple's ‘Magnifier’ and ‘SoundSight’ convey distances for PBLV – for either a single pixel, object or person, or for conveying an entire depth map respectively [7] .…”

“…When PBLV are interviewed about these tools, the most highly sought-after sensory information is distance for objects or people [6]. Recent iPhone Apps like 'LiDAR Sense', 'Super Lidar', Apple's 'Magnifier' and 'SoundSight' convey distances for PBLV -for either a single pixel, object or person, or for conveying an entire depth map respectively [7]. There are now multiple ways to gather distance information on modern smartphones, each technique impacting spatial accuracy and usability differently.…”

Sonified Distance in Sensory Substitution Does Not Always Improve Localization: Comparison With a 2-D and 3-D Handheld Device

“…Early SSVA devices encode 2D camera images into sounds and have shown potential for visual rehabilitation [3]- [8]. A new generation of SSVA devices takes advantage of the rapid development of 3D sensors to explicitly sonify depth [9]- [22].…”

“…Others use the repetition rate of beeps 2 to encode the distance of pedestrians in front of users [19] or to warn them of nearby obstacles [20]. Finally, some systems use reverberation to represent the depth of objects [21] or to inform users about the size of a room [22].…”