Auditory augmented reality: Object sonification for the visually impaired

Ribeiro, Flavio; Florêncio, Dinei; Chou, Philip A.; Zhang, Zhengyou

doi:10.1109/mmsp.2012.6343462

Cited by 67 publications

(39 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, they fail to return an efficient depth map in sunny environments because projected speckles are submerged by sunlight. As a result, approaches for VIP with light-coding sensors are just proof-of-concepts or only feasible in indoor environments [7,8,9,10,11,12,13,14,15]. …”

Section: Introductionmentioning

confidence: 99%

Expanding the Detection of Traversable Area with RealSense for the Visually Impaired

Yang

Wang

et al. 2016

Sensors

View full text Add to dashboard Cite

The introduction of RGB-Depth (RGB-D) sensors into the visually impaired people (VIP)-assisting area has stirred great interest of many researchers. However, the detection range of RGB-D sensors is limited by narrow depth field angle and sparse depth map in the distance, which hampers broader and longer traversability awareness. This paper proposes an effective approach to expand the detection of traversable area based on a RGB-D sensor, the Intel RealSense R200, which is compatible with both indoor and outdoor environments. The depth image of RealSense is enhanced with IR image large-scale matching and RGB image-guided filtering. Traversable area is obtained with RANdom SAmple Consensus (RANSAC) segmentation and surface normal vector estimation, preliminarily. A seeded growing region algorithm, combining the depth image and RGB image, enlarges the preliminary traversable area greatly. This is critical not only for avoiding close obstacles, but also for allowing superior path planning on navigation. The proposed approach has been tested on a score of indoor and outdoor scenarios. Moreover, the approach has been integrated into an assistance system, which consists of a wearable prototype and an audio interface. Furthermore, the presented approach has been proved to be useful and reliable by a field test with eight visually impaired volunteers.

show abstract

Section: Introductionmentioning

confidence: 99%

Expanding the Detection of Traversable Area with RealSense for the Visually Impaired

Yang

Wang

et al. 2016

Sensors

View full text Add to dashboard Cite

show abstract

“…Some components of what we implemented can be found in prior work (Botezatu et al, 2017; Ribeiro et al, 2012; Wang et al, 2017). Generally assistive devices have been designed to perform one well-circumscribed function, such as obstacle avoidance or route finding (Loomis et al, 2012; Roentgen et al, 2008).…”

Section: Discussionmentioning

confidence: 99%

Augmented Reality Powers a Cognitive Prosthesis for the Blind

Nrb

Meister

2018

Preprint

View full text Add to dashboard Cite

A non-invasive prosthesis for blind people endows objects in the environment with voices, 9 allowing a user to explore the scene, localize objects, and navigate through a building with 10 minimal training. 12 To restore vision for the blind several prosthetic approaches have been explored that convey raw 13 images to the brain. So far these schemes all suffer from a lack of bandwidth and the extensive 14 training required to interpret unusual stimuli. Here we present an alternate approach that restores 15 vision at the cognitive level, bypassing the need to convey sensory data. A wearable computer 16 captures video and other data, extracts the important scene knowledge, and conveys that through 17 auditory augmented reality. This system supports many aspects of visual cognition: from 18 obstacle avoidance to formation and recall of spatial memories, to long-range navigation. Neither 19 training nor modification of the physical environment are required: Blind subjects can navigate 20 an unfamiliar multi-story building on their first attempt. The combination of unprecedented 21 2 computing power in wearable devices with augmented reality technology promises a new era of 22 non-invasive prostheses that are limited only by software. 23 11 Abstract

show abstract

“…gain-control) is also limited in expressive sound quality, despite having a continuous response. Sensor signals are multiplied by a waveform of a chosen frequency, to "hear" vibrations [20,27,28]. This 1 There is no need here to compare synthesis methods such as ring modulation or FM synthesis, which merely generate artificial sounds, but do not themselves suggest how to decompose and remap arbitrary spectral regions of an input signal, to different arbitrary spectral regions.…”

Section: B Previous Transformation Methods For Sonificationmentioning

confidence: 99%

Swarm modulation: An algorithm for real-time spectral transformation

Janzen

Mann

2015

2015 IEEE Games Entertainment Media Conference (GEM)

View full text Add to dashboard Cite

A novel class of modulation is introduced, for frequency transformation and spectral synthesis in real-time. Swarm modulation has potential applications to enhance human hearing with extended frequency ranges, in medical diagnostics for electrocardiogram (ECG), electroencephalogram (EEG) and other medical signals, for RADAR analysis, for user-interface sonification, and for sound synthesis or non-synthetic sound transformation. Swarm modulation is a new way to transform signals, and is demonstrated for transforming subsonic and ultrasonic sound into the audible range of human hearing.Swarm modulation is based on the principle of phaseincoherent frequency-roaming oscillation. Features in the frequency-time plane are reconstructed via a time-varying process, controllable with instantaneous zero-latency reaction time to new information. Swarm modulation allows prioritization of salient output spectral features for efficient processing, and overcomes cyclic beating patterns when Fourier and waveletbased methods are applied in a stationary manner.Swarm modulation can flexibly re-map sound when a user expressively touches physical matter creating vibration. By detecting subsonic, sonic and ultrasonic vibrations, we can add to materials a rich acoustic user-feedback that can be adjusted to sound like a bell, xylophone, dull piece of wood, or a variety of other objects, in real-time. By dynamically controlling the output sound spectrum depending on the input spectrum, simultaneously with a continuous and low-latency temporal response, the system imitates the physicality of touching a real object.Applied in control panels and expressive control surfaces, swarm modulation can create realistic sonic feedback, for human head-up operation of controls in critical applications.

show abstract

Auditory augmented reality: Object sonification for the visually impaired

Cited by 67 publications

References 12 publications

Expanding the Detection of Traversable Area with RealSense for the Visually Impaired

Expanding the Detection of Traversable Area with RealSense for the Visually Impaired

Augmented Reality Powers a Cognitive Prosthesis for the Blind

Swarm modulation: An algorithm for real-time spectral transformation

Contact Info

Product

Resources

About