The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms

Abstract

Sensory substitution is a promising technique for mitigating the loss of a sensory modality. Sensory substitution devices (SSDs) work by converting information from the impaired sense (e.g., vision) into another, intact sense (e.g., audition). However, there are a potentially infinite number of ways of converting images into sounds, and it is important that the conversion takes into account the limits of human perception and other user-related factors (e.g., whether the sounds are pleasant to listen to). The device explored here is termed “polyglot” because it generates a very large set of solutions. Specifically, we adapt a procedure that has been in widespread use in the design of technology but has rarely been used as a tool to explore perception—namely, interactive genetic algorithms. In this procedure, a very large range of potential sensory substitution devices can be explored by creating a set of “genes” with different allelic variants (e.g., different ways of translating luminance into loudness). The most successful devices are then “bred” together, and we statistically explore the characteristics of the selected-for traits after multiple generations. The aim of the present study is to produce design guidelines for a better SSD. In three experiments, we vary the way that the fitness of the device is computed: by asking the user to rate the auditory aesthetics of different devices (Experiment 1), and by measuring the ability of participants to match sounds to images (Experiment 2) and the ability to perceptually discriminate between two sounds derived from similar images (Experiment 3). In each case, the traits selected for by the genetic algorithm represent the ideal SSD for that task. Taken together, these traits can guide the design of a better SSD.

EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution

“…In soundscapes generated from images, in contrast to western music, there is a tendency for adjacent frequencies to be played together. Using a semitone western scale would then generate sounds that are perceived as highly dissonant (Wright et al, 2013). Therefore, to generate more pleasant soundscapes, we used the pentatonic musical scale, which is a subset of the western musical scale (See supplementary materials) that generates less dissonance when adjacent notes are played together.…”

“…This was done because sounds with high energy in the 2500-5000 Hz range have been linked with unpleasantness (Kumar et al, 2008). In addition, a recent experiment investigating the perceived pleasantness of various SSD frequency ceilings showed that a ceiling of 2,500 Hz was selected as generating more pleasant soundscapes and a ceiling of 10,000 Hz was found to generate unpleasant soundscapes (Wright et al, 2013).…”

The Skin as a Medium for Sensory Substitution

“…Rather, I simply wish to question the choice of the tactile modality as the pre ferred mode of information transfer. Indeed, I remain much more opti mistic about sensory substitution and augmentation systems that utilize the auditory modality (e.g., Meijer, 1993;Wright and Ward, 2013). Here I would like to concur with the line taken by Loomis et al (2012) regarding the very modest progress seen when it comes to general-purpose sensory substitution systems when compared to the much more successful uptake of specific-purpose sensory substitution devices.…”

Sonic branding: A narrative review at the intersection of art and science

“…One might imagine a future in which AI could dynamically adjust the sonic expression of a brand to fit a listening environment or context, personalizing the advertising experience of a consumer (Maroely & Munichor, 2023; Suchman, 2024), or the potential for genetic algorithms to participate in the creation of sonic assets built on the crossmodal correspondences (cf. Wright & Ward, 2013). It has also been suggested that computational models may soon be able to predict perceived musical expression in branding scenarios (Lepa et al, 2020).…”