Opportunities for Human-AI Collaboration in Remote Sighted Assistance

Helping Helpers: Supporting Volunteers in Remote Sighted Assistance with Augmented Reality Maps

“…Although the expertise of trained agents leads to more robust and preferable assistance, limited access to users' visual feld through the smartphone's camera and location information can pose a substantial challenge to all agents, regardless of their training [51,63]. To address these challenges, the current generation of RSA services leveraged Google Maps and GPS location and sometimes utilized a pair of specialized glasses with wide-angle lenses for the users to increase their visual feld [50].…”

“…These measures enable agents to deliver directional information preemptively and subsequently improve the user experience in outdoor navigation [61]. However, providing navigational assistance indoors remains a key challenge for the agents for several reasons, such as weaker GPS signal, scarcity of indoor maps with fner details, and agents' lack of familiarity with the users' current physical environment [63,91]. In addition, researchers have identifed several open challenges for the agents common to providing navigational assistance outdoors and indoors.…”

“…In addition, researchers have identifed several open challenges for the agents common to providing navigational assistance outdoors and indoors. These challenges include the agent's difculty in continuously tracking and orienting the users on the map and within their (users') surroundings; estimating objects' depth, detecting landmarks and obstacles in users' camera feed; and interpreting and delivering the visual information in real-time via conversations [51,63]. Unfortunately, the current generation of RSA services is limited in addressing these challenges, which, however, impact the service quality of the agents, especially agents who are untrained or work as volunteers.…”

“…To address the above challenges, our prior work has envisioned assisting agents with computer vision (CV) technologies, particularly 3D map construction, object annotations, real-time localization, and video stream augmentation [33,63,91]. We also designed a series of low-fdelity prototypes (e.g., pictorials) illustrating how CV technologies can be adopted in RSA services [91].…”

Feasibility of Interactive 3D Map for Remote Sighted Assistance

“…As RSA services have gained popularity, a growing number of researchers started to study different aspects of these services. Kamikubo et al [26] and Lee et al [27,19,28,29] studied the interaction paradigm between RSA agents and visually impaired users and reported several challenges for RSA agents. Some of the key challenges include agents' lack of confidence due to unfamiliarity of the users' current physical environment, not having indoor maps with finer details, continuously tracking and orienting the user in the map and within his or her surroundings, estimating objects' depth in users' camera feed and describing those in realtime, and developing mutual trust and synergy with the users.…”