Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents

Tani, Jacopo; Daniele, Andrea F.; Bernasconi, Gianmarco; Camus, Amaury; Petrov, Aleksandar; Courchesne, Anthony; Mehta, Bhairav; Suri, Rohit; Zaluska, Tomasz; Walter, Matthew R.; Paull, Liam; Censi, Andrea

doi:10.1109/iros45743.2020.9341677

Cited by 9 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several works have recently proposed benchmarking frameworks for evaluating robot motion planning algorithms from the classical navigation perspective [376][377][378][379][380][381][382][383][384][385], i.e., without considering human awareness constraints. These works mainly focus on performance metrics like navigation success rate, path length, or time required to reach the goal.…”

Section: Simulatorsmentioning

confidence: 99%

Bridging Requirements, Planning, and Evaluation: A Review of Social Robot Navigation

Karwowski,

Szynkiewicz,

Niewiadomska-Szynkiewicz

2024

Sensors

View full text Add to dashboard Cite

Navigation lies at the core of social robotics, enabling robots to navigate and interact seamlessly in human environments. The primary focus of human-aware robot navigation is minimizing discomfort among surrounding humans. Our review explores user studies, examining factors that cause human discomfort, to perform the grounding of social robot navigation requirements and to form a taxonomy of elementary necessities that should be implemented by comprehensive algorithms. This survey also discusses human-aware navigation from an algorithmic perspective, reviewing the perception and motion planning methods integral to social navigation. Additionally, the review investigates different types of studies and tools facilitating the evaluation of social robot navigation approaches, namely datasets, simulators, and benchmarks. Our survey also identifies the main challenges of human-aware navigation, highlighting the essential future work perspectives. This work stands out from other review papers, as it not only investigates the variety of methods for implementing human awareness in robot control systems but also classifies the approaches according to the grounded requirements regarded in their objectives.

show abstract

Section: Simulatorsmentioning

confidence: 99%

Bridging Requirements, Planning, and Evaluation: A Review of Social Robot Navigation

Karwowski,

Szynkiewicz,

Niewiadomska-Szynkiewicz

2024

Sensors

View full text Add to dashboard Cite

show abstract

“…Section V) as well as the experimental procedure (study design, questionnaires etc.) available [281]. These issues are addressed by initiatives to foster publications with extended, detailed information on the used hardware and software implementation: So-called 'R-Articles' must be accompanied by mandatory, in-depth system descriptions, code, and further data relevant for reproducing experimental setups [282].…”

Section: ) Reproducibilitymentioning

confidence: 99%

Benchmarking Teamwork of Humans and Cobots—An Overview of Metrics, Strategies, and Tasks

2023

View full text Add to dashboard Cite

Human-robot teaming receives an ever-increasing level of attention in research, development and industry. Novel approaches to task sharing in hybrid teams range from optimized schedules to intelligent cobot assistants with a high degree of autonomy. These approaches must prove their usefulness and benefits compared to manual work or full automation -particularly when it comes to assessing their potential for productive industrial use. This leverages demand for standardized, repeatable benchmarks to compare approaches and measure improvements in a reproducible way. Designing such benchmarks is challenging as numerous aspects, from safety considerations to human factors and team performance, must be considered. This survey seeks to contribute to the future development of benchmarks for the field of collaborative assembly, handling, and industrial cobot applications by giving a comprehensive overview of relevant metrics, evaluation strategies, and tasks for human-robot teams.

show abstract

“…Due to a growing set of navigation algorithms available, the importance of quantitative evaluation has increased. Several authors have recently proposed benchmarking frameworks for evaluating robot motion planning algorithms [4], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18].…”

Section: Related Workmentioning

confidence: 99%

Quantitative Metrics for Benchmarking Human-Aware Robot Navigation

Karwowski

Szynkiewicz

2023

IEEE Access

View full text Add to dashboard Cite

Social robots have recently gained popularity, and many human-aware navigation approaches have emerged. This work presents a comprehensive benchmark for quantitatively assessing robot navigation methods. As an automated quantitative approach, our Social Robot Planner Benchmark (SRPB) produces invariable indicators of the algorithm's performance that can assist the system designer in selecting the best method for the specific application. Our benchmark extends state-of-the-art task performance scores and proposes novel social metrics regarding robot motion naturalness and the perceived safety of humans surrounding the robot. Our social metrics take human tracking reliability into account. Using the SRPB integrated with the TIAGo robot, we assessed the robot's behaviour operating with traditional and human-aware trajectory planners in simulated and real-world environments. Our experiments tested whether state-ofthe-art human-aware trajectory planners significantly improve human-awareness indicators over traditional approaches yet still maintain reasonable navigation performance. An open-source implementation of our benchmark, compatible with the Robot Operating System, is provided.

show abstract

Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents

Cited by 9 publications

References 20 publications

Bridging Requirements, Planning, and Evaluation: A Review of Social Robot Navigation

Bridging Requirements, Planning, and Evaluation: A Review of Social Robot Navigation

Benchmarking Teamwork of Humans and Cobots—An Overview of Metrics, Strategies, and Tasks

Quantitative Metrics for Benchmarking Human-Aware Robot Navigation

Contact Info

Product

Resources

About