Benchmarking Protocol for Grasp Planning Algorithms

Abstract

Numerous grasp planning algorithms have been proposed since the 1980s. The grasping literature has expanded rapidly in recent years, building on greatly improved vision systems and computing power. Methods have been proposed to plan stable grasps on: known objects (exact 3D model is available), familiar objects (e.g. exploiting a-priori known grasps for different objects of the same category), or novel object shapes observed during task execution. Few of these methods have ever been compared in a systematic way, and objective performance evaluation of such complex systems remains problematic. Difficulties and confounding factors include: different assumptions and amounts of a-priori knowledge in different algorithms; different robots, hands, vision systems and setups in different labs; different choices or application needs for grasped objects. Also, grasp planning can use different grasp quality metrics (including empirical or theoretical stability measures), or other criteria, e.g. computational speed, or combination of grasps with reachability considerations. While acknowledging and discussing the outstanding difficulties surrounding this complex topic, we propose a methodology for reproducible experiments to compare the performance of a variety of grasp planning algorithms. Our protocol attempts to improve the objectivity with which different grasp planners are compared by minimising the influence of key components in the grasping pipeline, e.g. vision and pose estimation. The protocol is demonstrated by evaluating two different grasp planners: a state-of-the-art model-free planner, and a popular open-source model-based planner. We show results from real-robot experiments with a 7-DoF arm and 2-finger hand, and simulation-based evaluations.

Aiding Grasp Synthesis for Novel Objects Using Heuristic-Based and Data-Driven Active Vision Methods

“…The real world testing was done on a subset of objects in simulation along with two custom objects built using lego pieces. The grasp benchmarking protocol in (Bekiroglu et al, 2020) was implemented to assess the grasp quality based on the five scoring parameters specified. Another grasp benchmarking protocol focused on visionbased approaches is Kootstra et al (2012), but it is simulation based and needs the input to be in the form of stereo images which does not fit well with our pipeline's need for a point cloud input.…”

“…Another grasp benchmarking protocol focused on visionbased approaches is Kootstra et al (2012), but it is simulation based and needs the input to be in the form of stereo images which does not fit well with our pipeline's need for a point cloud input. Also, it lacks the shaking and rotational test metrics available in Bekiroglu et al (2020). The 3D Heuristic and the Q-Learning policies were selected and tested with the objects.…”

“…In the simulation, we created benchmarks to assess not only whether our policies do better than random but to measure how close each approach comes to optimal behavior for each object. In the real-world experiments, we have adopted an existing grasp planning benchmark (Bekiroglu et al, 2020), and assess how well the simulation performances translate to real systems.…”

Guest Editorial: Introduction to the Special Issue on Benchmarking Protocols for Robotic Manipulation

“…In [3], the benchmark identifies the limits and capabilities of the robotic system (e.g., workspace limits, payload limits) and allows to normalize the results accordingly. The benchmark in [4] provides a rigorous procedure to assess the performance of grasp planning algorithms, while minimizing the effects of other elements in the grasping pipeline. While [3], [4] are focused on grasping from table top, [5] provides a benchmark for a more industrial application, i.e., bin-picking, considering pick-andplace of fruit and vegetables.…”

PrendoSim: Proxy-Hand-Based Robot Grasp Generator

“…However, there is no consensus on how to define stability across the community. More operational definitions consider grasps to be stable whenever the object is held by the robot gripper for more than a certain amount of time, or after shaking (Bekiroglu et al, 2020). PrendoSim similarly adopts an operational definition of stability, which we will cover in detail in the following sections.…”