Dynamic Crosswalk Scene Understanding for the Visually Impaired

Abstract

Independent mobility poses a great challenge to the visually impaired individuals. This paper proposes a novel system to understand dynamic crosswalk scenes, which detects the key objects, such as crosswalk, vehicle, and pedestrian, and identifies pedestrian traffic light status. The indication of where and when to cross the road is provided to the visually impaired based on the crosswalk scene understanding. Our proposed system is implemented on a head-mounted mobile device (Sensin-gAI G1) equipped with an Intel RealSense camera and a cellphone, and provides surrounding scene information to visually impaired individuals through audio signal. To validate the performance of the proposed system, we propose a crosswalk scene understanding dataset which contains three sub-datasets: a pedestrian traffic light dataset with 7447 images, a dataset of key objects on the crossroad with 1006 images and a crosswalk dataset with 3336 images. Extensive experiments demonstrated that the proposed system was robust and outperformed the state-of-the-art approaches. The experiment conducted with the visually impaired subjects shows that the system is practical useful.

Side Collision Detection Model for Visually Impaired Using Monocular Object-Specific Distance Estimation and Multimodal Real-World Location Calculation

“…Li et al (2020) described a wearable application that uses an RGB-D camera and an inertial measurement unit (IMU) to identify objects in real-time and create collision-free routes to assist visually impaired persons in swiftly gaining situational awareness and walking safely. Tian et al (2021) combined the depth camera and object recognition, realizing the detection and early warning of obstacles. And they utilize neural network training to identify traffic light signals and sidewalks, which gives users more road condition information.…”

“…However, considering the cost, we opt for a monocular vision camera for data acquisition. For detection range, all the models take the forward direction into account and Tian et al (2021) stress the risk of the vehicle forward. Katzschmann et al (2018) sets TOF sensors at each side of the sensor belt to detect possible obstacles like branches.…”

Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution

“…The objective assessment of the perceptual experience of visual information from human behavioral observations is another aspect that needs to be seriously considered in visualauditory sensory substitution. Many previous studies have investigated the perceptual experience of sensory substitution based on behavioral observations of the participants [19]- [21], [24], [25]. The participants are asked to perform tasks to extract visual information from a substituted audio signal.…”

From Pedestrian Detection to Crosswalk Estimation: An EM Algorithm and Analysis on Diverse Datasets

“…Past approaches to detecting crosswalks have been primarily based on camera images of the static scene. Such methods include figure-ground segmentation (Coughlan & Shen, 2006), edge direction and cross-ratio analysis (Suzuki et al, 2010), Canny edge detection and Hough line parameter estimation (Tian et al, 2021), MSER and ERANSAC (Zhai et al, 2015), laser scanning (Hernndez et al, 2015), monocular image deep learning to classify presence of crosswalk in overall scene (Tmen & Ergen, 2020), PCA-based road segmentation and crosswalk detection from a LiDAR point cloud (Riveiro et al, 2015), and image classification using deep learning over massive instances from satellite imagery (Berriel et al, 2017).…”