Can We Unify Perception and Localization in Assisted Navigation? An Indoor Semantic Visual Positioning System for Visually Impaired People

Chen, Haoye; Zhang, Yingzhi; Yang, Kailun; Martínez, Manuel Martínez; Müller, Karin; Stiefelhagen, Rainer

doi:10.1007/978-3-030-58796-3_13

Cited by 9 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work is concerned about the lightweight network to process scenarios captured by sensors for more efficient computing and processing capabilities. The model is deployed on a wearable system composed of a Realsense L515 and an Nvidia Jetson AGX Xavier processor so that the wearable system is able to accompany visually impaired people and assist them with acoustic feedback, e.g., by combining the system with bone-conduction earphones on the wearable glasses [10]. A comprehensive set of experiments on both public datasets and real-world scenes captured by our wearable system, demonstrates the effectiveness of the presented universal perception framework.…”

Section: Introductionmentioning

confidence: 90%

“…As a multi-sensor system, SurImage Stereo Camera complements the RealSense in various conditions. system [10] with a pair of smart glasses informs the visually impaired people based on semantic segmentation and 3D reconstruction. Differently, our work puts the focus on the multi-task model and satisfying the real-time requirement simultaneously.…”

Section: Related Work a Visual Assistive Systemmentioning

confidence: 99%

See 1 more Smart Citation

Perception Framework through Real-Time Semantic Segmentation and Scene Recognition on a Wearable System for the Visually Impaired

Zhang¹,

Chen²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

As the scene information, including objectness and scene type, are important for people with visual impairment, in this work we present a multi-task efficient perception system for the scene parsing and recognition tasks. Building on the compact ResNet backbone, our designed network architecture has two paths with shared parameters. In the structure, the semantic segmentation path integrates fast attention, with the aim of harvesting long-range contextual information in an efficient manner. Simultaneously, the scene recognition path attains the scene type inference by passing the semantic features into semantic-driven attention networks and combining the semantic extracted representations with the RGB extracted representations through a gated attention module. In the experiments, we have verified the systems' accuracy and efficiency on both public datasets and real-world scenes. This system runs on a wearable belt with an Intel RealSense LiDAR camera and an Nvidia Jetson AGX Xavier processor, which can accompany visually impaired people and provide assistive scene information in their navigation tasks.

show abstract

Section: Introductionmentioning

confidence: 90%

Section: Related Work a Visual Assistive Systemmentioning

confidence: 99%

Perception Framework through Real-Time Semantic Segmentation and Scene Recognition on a Wearable System for the Visually Impaired

Zhang¹,

Chen²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [6], a kinetic real-time CNN was customized for the recognition of road barriers to support navigation assistance, which are usually set at the gate of a residential area or working area. The wearable system [7] with a pair of smart glasses informs the visually impaired people based on semantic segmentation and 3D reconstruction. In [8], a wearable belt is developed with deep learning system to pinpoint the exact locations of surrounding objects and the scene type in real-time.…”

Section: Related Workmentioning

confidence: 99%

ViT Cane: Visual Assistant for the Visually Impaired

Kumar

2021

Preprint

View full text Add to dashboard Cite

Blind and visually challenged face multiple issues with navigating the world independently. Some of these challenges include finding the shortest path to a destination and detecting obstacles from a distance. To tackle this issue, this paper proposes ViT Cane, which leverages a vision transformer model in order to detect obstacles in real-time.Our entire system consists of a Pi Camera Module v2, Raspberry Pi 4B with 8GB Ram and 4 motors. Based on tactile input using the 4 motors, the obstacle detection model is highly efficient in helping visually impaired navigate unknown terrain and is designed to be easily reproduced. The paper discusses the utility of a Visual Transformer model in comparison to other CNN based models for this specific application. Through rigorous testing, the proposed obstacle detection model has achieved higher performance on the Common Object in Context (COCO) data set than its CNN counterpart. Comprehensive field tests were conducted to verify the effectiveness of our system for holistic indoor understanding and obstacle avoidance.

show abstract

“…The prominence of mobile computing has been increasing with the emergence of devices like wearable smart glasses, small camera-based sensors, and compact computers. For example, there are many applications that can benefit from semantic information such as robots working in a factory, extended reality applications, drones [45], [46], [51], [53], and wearable system exploiting scene understanding [1], [5], [7], [17], [30], [31], [49], [50]. These systems process images or videos in real-time to estimate pose and guide movement.…”

Section: Introductionmentioning

confidence: 99%

Semantic Segmentation Optimized for Low Compute Embedded Devices

Son

Weiland

2022

IEEE Access

View full text Add to dashboard Cite

Deployment of a deep convolutional neural network on a low compute device is an increasingly important area of research. Wearable systems, drones, and robots use semantic information to manipulate, control, and estimate variables related to efficient navigation or gain of contextual information. However, real-time semantic segmentation is challenging on low compute devices. We propose a compact convolutional neural network for real-time applications on a low compute device. Our decoder uses pixel shuffling to achieve efficient inferences. We compared our CNN with state-of-the-art models ranked on the Cityscapes real-time semantic segmentation category. We propose a modified Net score that includes frame per seconds as an additional metric along with traditional metrics mIoU, GFLOPs, and the number of parameters to evaluate mobile computing performance. Our CNN achieved 65.7 FPS, 76.7% mIoU without ImageNet pre-training, while requiring 25 GFLOPs and 4.55M parameters, resulting in a 127.53 on the modified Net score compared to 119.89 for DDRNET23_slim and 115.39 for Regseg. In addition to Cityscapes results, our CNN performance (83.3% mIoU and 354 FPS with TensorRT) was superior to published mIoU values for Regseg and other CNNs on Camvid test set. This performance achieves the state-of-the-art. To demonstrate compatibility with low compute devices, we evaluated our CNN on two mobile computing platforms and showed real-time performance (57 fps) on Jetson NX 8 GB with TensorRT and 12.65 fps on Jetson Xavier AGX without TensorRT. Our CNN can operate with high accuracy on low compute devices to support system which benefit from semantic information. INDEX TERMS Semantic segmentation, hardware-constrained, low compute devices, real-time performance, deep convolutional neural network, Jetson Xavier AGX, Jetson Nx.

show abstract

Can We Unify Perception and Localization in Assisted Navigation? An Indoor Semantic Visual Positioning System for Visually Impaired People

Cited by 9 publications

References 18 publications

Perception Framework through Real-Time Semantic Segmentation and Scene Recognition on a Wearable System for the Visually Impaired

Perception Framework through Real-Time Semantic Segmentation and Scene Recognition on a Wearable System for the Visually Impaired

ViT Cane: Visual Assistant for the Visually Impaired

Semantic Segmentation Optimized for Low Compute Embedded Devices

Contact Info

Product

Resources

About