BlindPilot: A Robotic Local Navigation System that Leads Blind People to a Landmark Object

The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering by forcing the features to be closer together in the feature space. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach. INDEX TERMSSelf-supervised, Terrain type clustering, Multi-modal learning Training Testing OR FIGURE 1: Overview of our terrain clustering framework.We train the model to extract the features from audio-visual data in a self-supervised manner. At the testing, we assume that only a single modality (either audio or visual) can be accessed due to the extreme conditions, the obtained data is incrementally clustered into terrain types.

show abstract

“…+ M SE(w(x raw , x edge , x audio ), w(x audio )), (7) where w is determined as the non-reparametarized vector calculated with µ(x) and σ(x):…”

Section: E Additional Loss Termmentioning

confidence: 99%

“…Similarly, information about the surrounding terrain may help a robot modify its course of action during autonomous navigation [2], [6]. In the field of assistive robotics [7], the robot can warn a visually impaired person of potential danger concerning the ground type [8], [9].…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Audio-Visual Feature Learning for Single-Modal Incremental Terrain Type Clustering

2021

View full text Add to dashboard Cite

show abstract

“…The size of the ground field, where pedestrians are walking from starting points to their destinations, is configured to be 20m × 40m. We spawned 50 pedestrians starting from one of the four corners of the field, [−10, −10], [10,10], [10, −20], [10,20], and set the opposite side of the field as their destinations. Both the starting points and destinations were randomized with a uniform distribution.…”

Section: B Dataset Details B1 Example Trajectoriesmentioning

confidence: 99%

“…Our contributions are threefold: (i) the introduction of view birdification, the simultaneous recovery of ground-plane trajectories of surrounding pedestrians and that of the observation camera just from an ego-centric view, as a novel research problem, (ii) the deriva-tion of a cascaded optimization framework with a Bayesian formulation to solve the view birdification problem, and (iii) the construction of view birdification datasets consisting of paired real human trajectories and synthetic ego-views. We believe view birdification finds a wide range of applications and these contributions have strong implications in computer vision and robotics as they establish view birdification as a foundation for downstream visual understanding applications including crowd behavior analysis [1,2,12], self-guidance [19,20], and robot navigation [32,39].…”

Section: Introductionmentioning

confidence: 99%

View Birdification in the Crowd: Ground-Plane Localization from Perceived Movements

Nishimura¹,

Nobuhara²,

Nishino³

2021

Preprint

View full text Add to dashboard Cite

We introduce view birdification, the problem of recovering ground-plane movements of people in a crowd from an ego-centric video captured from an observer (e.g., a person or a vehicle) also moving in the crowd. Recovered ground-plane movements would provide a sound basis for situational understanding and benefit downstream applications in computer vision and robotics. In this paper, we formulate view birdification as a geometric trajectory reconstruction problem and derive a cascaded optimization method from a Bayesian perspective. The method first estimates the observer's movement and then localizes surrounding pedestrians for each frame while taking into account the local interactions between them. We introduce three datasets by leveraging synthetic and real trajectories of people in crowds and evaluate the effectiveness of our method. The results demonstrate the accuracy of our method and set the ground for further studies of view birdification as an important but challenging visual understanding problem.

show abstract

“…Cabot received high praise from BLV users on the feeling of comfort, confidence, and safety. Kayukawa et al [29] proposed another car-form navigation robot that walked on the right side of users, which had consistency in selecting sensors and processors with the former robot. They assembled a LiDAR, stereo camera, and laptop, which could hardly be implemented on a white cane.…”

Section: Currently Grounded Robots With Different Machine Formsmentioning

confidence: 99%

Reproducible abnormalities of functional gradient reliably predict clinical and cognitive symptoms in schizophrenia

Wang

Liu

et al. 2020

Preprint

View full text Add to dashboard Cite

BackgroundSchizophrenia (SZ) typically manifests heterogeneous phenotypes involving positive, negative and cognitive symptoms. However, the underlying neural mechanisms of these symptoms keep unclear. Functional gradient is a fascinating measure to characterize continuous, hierarchical organization of brain.MethodsWe aimed to investigate whether reproducible disruptions of functional gradient existed in SZ compared to normal controls (NC), and these abnormalities were associated with severity of clinical and cognitive symptoms in SZ. All analyses were implemented in two independent large-sample multi-site datasets (discovery dataset, 400 SZ and 336 NC; replication dataset, 279 SZ and 262 NC). First, functional gradient across cerebral cortex was calculated in each subject. Second, vertex-wise comparisons of cortical gradient between SZ and NC groups were performed to identify abnormalities in SZ. Meanwhile, reproducible and robustness analyses were implemented to validate these abnormalities. Finally, regression analyses were performed using generalized additive models to link these abnormalities to severity of clinical and cognitive symptoms in SZ.ResultsWe found an abnormal gradient map in SZ in the discovery dataset, which was reproducible in the replication dataset. The abnormal gradient pattern was also robust when performing methodological alternatives and control analyses. Further, these reproducible abnormalities can reliably predict symptoms of clinical and cognitive domains across the two independent datasets.ConclusionThese findings demonstrated that alterations in functional gradient can provide a reliable signature of SZ, characterizing the heterogenous symptoms of clinical or cognitive domains, and may be further investigated to understand the neurobiological mechanisms of these symptoms.Impact StatementIn our study, using functional gradient measure and statistical learning technology and two independent multi-site case-control resting-state fMRI datasets (discovery dataset: 736 subjects; replication dataset: 541 subjects), we comprehensively investigated functional hierarchical organization in the cerebral cortex of SZ and its association with interindividual severity of symptoms. We found reproducible and robust abnormalities of functional gradient existed in SZ, which provided a reliable signature to characterize negative and general psychopathology symptoms, as well as cognitive deficits. Our findings can provide new insights to understand the neurobiological mechanisms of clinical and cognitive symptoms in SZ.

show abstract

BlindPilot: A Robotic Local Navigation System that Leads Blind People to a Landmark Object

Cited by 24 publications

References 32 publications

Self-Supervised Audio-Visual Feature Learning for Single-Modal Incremental Terrain Type Clustering

Self-Supervised Audio-Visual Feature Learning for Single-Modal Incremental Terrain Type Clustering

View Birdification in the Crowd: Ground-Plane Localization from Perceived Movements

Reproducible abnormalities of functional gradient reliably predict clinical and cognitive symptoms in schizophrenia

Contact Info

Product

Resources

About