A System for Assisting the Visually Impaired in Localization and Grasp of Desired Objects

Aldas

ACM Trans. Access. Comput.

et al. 2022

Locating and grasping objects is a critical task in people’s daily lives. For people with visual impairments, this task can be a daily struggle. The support of augmented reality frameworks in smartphones can overcome the limitations of current object detection applications designed for people with visual impairments. We present AIGuide, a self-contained smartphone application that leverages augmented reality technology to help users locate and pick up objects around them. We conducted a user study to investigate the effectiveness of AIGuide in a visual prosthetic for providing guidance; compare it to other assistive technology form factors; investigate the use of multimodal feedback, and provide feedback about the overall experience. We gathered performance data and participants’ reactions and analyzed videos to understand users’ interactions with the nonvisual smartphone user interface. Our results show that AIGuide is a promising technology to help people with visual impairments locate and acquire objects in their daily routine. The benefits of AIGuide may be enhanced with appropriate interaction design.

Section: Related Work 21 Object Finding Approachesmentioning

confidence: 99%

Section: Hand Guidancementioning

confidence: 99%

AIGuide: Augmented Reality Hand Guidance in a Visual Prosthetic

Aldas

ACM Trans. Access. Comput.

et al. 2022

Proceedings of the 25th International Conference on Auditory Display (ICAD 2019)

“…Some systems have instead utilized Text-To-Speech (TTS) to describe the location of nearby objects. Thakoor et al [9] tested The 25 th International Conference on Auditory Display (ICAD 2019) 23-27 June 2019, Northumbria University a system that provided TTS denoting the presence of objects recognized by a mobile camera in one of nine areas in front of the user (e.g. "upper right").…”

Section: The Sound Of Spacementioning

confidence: 99%

Auditory Displays to Facilitate Object Targeting in 3D Space

May¹,

Sobel²,

Wilson³

et al. 2019

In both extreme and everyday situations, humans need to find nearby objects that cannot be located visually. In such situations, auditory display technology could be used to display information supporting object targeting. Unfortunately, spatial audio inadequately conveys sound source elevation, which is crucial for locating objects in 3D space. To address this, three auditory display concepts were developed and evaluated in the context of finding objects within a virtual room, in either low or no visibility conditions: (1) a one-time height-denoting “area cue,” (2) ongoing “proximity feedback,” or (3) both. All three led to improvements in performance and subjective workload compared to no sound. Displays (2) and (3) led to the largest improvements. This pattern was smaller, but still present, when visibility was low, compared to no visibility. These results indicate that persons who need to locate nearby objects in limited visibility conditions could benefit from the types of auditory displays considered here.

“…Recently, many aid systems based on computer vision and image processing algorithms have been introduced and developed to meet the visually impaired and blind people's needs with scene understanding and interaction. However, most of the proposed aid systems, even traditional ones such as white canes and dog assistants, have been provided to facilitate navigation [1,2,3,4], obstacle avoidance, localisations, and grasping objects [5], without considering interactions with the surrounding people which represent a huge part of our daily activities. visually impaired and blind people needs to have an impression of human actions in order to be able to interact with them and get along with society.…”

Section: Introductionmentioning

confidence: 99%

Semantic Labeling of Human Action For Visually Impaired And Blind People Scene Interaction

Leyla¹,

Larabi²

2022

Preprint

The aim of this work is to contribute to the development of a tactile device for visually impaired and blind persons in order to let them to understand actions of the surrounding people and to interact with them. First, based on the state-of-the-art methods of human action recognition from RGB − D sequences, we use the skeleton information provided by Kinect, with the disentangled and unified multi-scale Graph Convolutional (M S − G3D) model to recognize the performed actions. We tested this model on real scenes and found some of constraints and limitations. Next, we apply a fusion between skeleton modality with M S − G3D and depth modality with CNN in order to bypass the discussed limitations. Third, the recognized actions are labeled semantically and will be mapped into an output device perceivable by the touch sense.