Benchmarking In-Hand Manipulation

Abstract

The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states (i.e. static object poses and fingertip locations) for various in-hand manipulation tasks. We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across all tasks, also serves as a measure of the system's in-hand manipulation capability. We provide supporting software, task examples, and evaluation results associated with the benchmark.

Complex In-Hand Manipulation via Compliance-Enabled Finger Gaiting and Multi-Modal Planning

“…We provide benchmarking baselines for our experimentation in Fig. 11 according to guidelines outlined [33]. Please refer to the supplementary video for experimental evaluations.…”

Guest Editorial: Introduction to the Special Issue on Benchmarking Protocols for Robotic Manipulation

“…Four other letters in the special issue focus on more specific robotic manipulation tasks. In [11], a benchmark is provided for assessing in-hand manipulation accuracy. In [12], a benchmark is provided for assessing aerial manipulation capabilities, e.g., aerial grasping, force control etc.…”

MotionBenchMaker: A Tool to Generate and Benchmark Motion Planning Datasets

“…Vehicle Navigation CommonRoad [15] 2017 × × × × × Robot@Home [16] 2017 × × × × × Multi-Agent Path-Find Benchmark [17] 2019 × × × × × MAVBench [18] 2020 × × × × × BARN [19] 2020 × × × Bench-MR [20] 2021 × × × × PathBench [21] 2021 × × × General Robotics OMPLBenchmarks [22] 2015 × × × × × Robobench [23] 2016 × × Roboturk (Teleoperation database) [24] 2019 × × × RLBench [25] 2020 × OCRTOC [26] 2021 × Robot Manipulation ACRV picking benchmark [2] 2017 × × RoboNet [27] 2019 × × × GraspNet [28] 2020 × × × × × × Brown Planning Benchmarks [29] 2020 × × Aerial Manipulation [30] 2020 × × × Bimanual Manipulation Benchmark [31] 2020 × × In-hand manipulation benchmark [32] 2020 × × × × ProbRobScene [33] 2021…”

“…Similar datasets and benchmark utilities concentrate on specific aspects of manipulation. This involves tasks like bimanual manipulation [31], in-hand manipulation [32], cloth manipulation [38], aerial manipulation [30], or solving Rubik's cube [39].…”