The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research

Abstract

Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark. Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems -including perception and manipulation -instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.

Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach

“…The ACRV Picking Benchmark (APB) [16] and the YCB Object Set [5] define item sets and manipulation tasks, but benchmark on tasks such as warehouse order fulfilment (APB) or table setting and block stacking (YCB) rather than raw grasp success rate as is typically reported. Additionally, many of the items from these two sets are impractically small, large or heavy for many robots and grippers, so have not been widely adopted for robotic grasping experiments.…”

“…minimal objects with similar shapes). The objects were chosen from the standard robotic grasping datasets the ACRV Picking Benchmark (APB) [16] and the YCB Object Set [5], both of which provide item specifications and online purchase links. Half of the item classes (mug, screwdriver, marker pen, die, ball and clamp) appear in both data sets.…”

Cartman: The Low-Cost Cartesian Manipulator that Won the Amazon Robotics Challenge

“…In previous years' challenges, as this one, almost all teams have competed using articulated robotic arms [1]. We had previously competed with a Baxter research robot [3], and encountered difficulties in planning movements of its 7-DoF arm within the confines of an Amazon storage shelf. Linear movement of an articulated arm requires control of multiple joints, which may result in sections of the arm colliding with the shelf.…”

GRASPA 1.0: GRASPA is a Robot Arm graSping Performance BenchmArk

“…The ACRV benchmark [17] and the one published by Triantafyllou et al [18] tackle the issue of reproducibility by proposing a set of objects and layouts for industrial shelving and pick and place applications. Both argue that physical execution of the task is essential in evaluating the performance of pick and place pipelines, although their protocols do not account for test platform limitations and the score metrics do not provide insight on the performance of single pipeline steps.…”