SocNavBench: A Grounded Simulation Testing Framework for Evaluating Social Navigation

Biswas, Abhijat; Wang, Allan; Silvera, Gustavo; Steinfeld, Aaron; Admoni, Henny

doi:10.48550/arxiv.2103.00047

Cited by 4 publications

(8 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, they do not allow quantification of the robot's actions' impact on surrounding humans. Biswas et al [15] proposed a set of metrics for evaluating crowd navigation algorithms; nonetheless, their metrics remain mostly focused on robot efficiency rather than social conformity.…”

Section: Related Workmentioning

confidence: 99%

Metrics for Evaluating Social Conformity of Crowd Navigation Algorithms

Wang¹,

Chan²,

Carreno‐Medrano³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent protocols and metrics for training and evaluating autonomous robot navigation through crowds are inconsistent due to diversified definitions of "social behavior". This makes it difficult, if not impossible, to effectively compare published navigation algorithms. Furthermore, with the lack of a good evaluation protocol, resulting algorithms may fail to generalize, due to lack of diversity in training. To address these gaps, this paper facilitates a more comprehensive evaluation and objective comparison of crowd navigation algorithms by proposing a consistent set of metrics that accounts for both efficiency and social conformity, and a systematic protocol comprising multiple crowd navigation scenarios of varying complexity for evaluation. We tested four state-of-the-art algorithms under this protocol. Results revealed that some state-of-the-art algorithms have much challenge in generalizing, and using our protocol for training, we were able to improve the algorithm's performance. We demonstrate that the set of proposed metrics provides more insight and effectively differentiates the performance of these algorithms with respect to efficiency and social conformity.

show abstract

Section: Related Workmentioning

confidence: 99%

Metrics for Evaluating Social Conformity of Crowd Navigation Algorithms

Wang¹,

Chan²,

Carreno‐Medrano³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Simulators can fill in the role of datasets for both training and evaluation. Simulators such as PedSIM [15], CrowdNav [10], SocNavBench [4] and SEAN [42] are in use by the research community. However, sim-to-real transfer is an unsolved problem in robotics.…”

Section: B Simulators and Pedestrian Datasetsmentioning

confidence: 99%

Towards Rich, Portable, and Large-Scale Pedestrian Data Collection

Wang¹,

Biswas²,

Admoni³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, pedestrian behavior research has shifted towards machine learning based methods and converged on the topic of modeling pedestrian interactions. For this, a large-scale dataset that contains rich information is needed. We propose a data collection system that is portable, which facilitates accessible large-scale data collection in diverse environments. We also couple the system with a semi-autonomous labeling pipeline for fast trajectory label production. We demonstrate the effectiveness of our system by further introducing a dataset we have collected -the TBD pedestrian dataset. Compared with existing pedestrian datasets, our dataset contains three components: human verified labels grounded in the metric space, a combination of top-down and perspective views, and naturalistic human behavior in the presence of a socially appropriate "robot". In addition, the TBD pedestrian dataset is larger in quantity compared to similar existing datasets and contains unique pedestrian behavior.

show abstract

“…Three widely adopted metrics, deemed as important for socially navigating robots, were evaluated across the three experimental conditions: the Path Length Ratio (PLR), the Path Regularity (PR), and the Closest Pedestrian Distance (CPD) (46).…”

Section: Analysis Of Performance Metricsmentioning

confidence: 99%

“…The PLR is defined as the ratio between the length of the line-of-sight path between the initial and final point of a path and the actual path length between the same two points (46). A higher path length ratio is usually preferred, since it indicates that an agent minimizes the length of the path to reach its goal.…”

Section: Analysis Of Performance Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Game-theoretical trajectory planning enhances social acceptability for humans

Galati¹,

Primatesta²,

Grammatico³

et al. 2022

Preprint

View full text Add to dashboard Cite

Since humans and robots are increasingly sharing portions of their operational spaces, experimental evidence is needed to ascertain the safety and social acceptability of robots in human-populated environments. Although several studies have aimed at devising strategies for robot trajectory planning to perform safe motion in populated environments, a few efforts have measured to what extent a robot trajectory is accepted by humans. Here, we present a navigation system for autonomous robotics that ensures safety and social acceptability of robotic trajectories. We overcome the typical reactive nature of state-of-the-art trajectory planners by leveraging non-cooperative game theory to design a planner that encapsulates human-like features of preservation 1

show abstract

SocNavBench: A Grounded Simulation Testing Framework for Evaluating Social Navigation

Cited by 4 publications

References 43 publications

Metrics for Evaluating Social Conformity of Crowd Navigation Algorithms

Metrics for Evaluating Social Conformity of Crowd Navigation Algorithms

Towards Rich, Portable, and Large-Scale Pedestrian Data Collection

Game-theoretical trajectory planning enhances social acceptability for humans

Contact Info

Product

Resources

About