The Cluttered Environment Picking Benchmark (CEPB) for Advanced Warehouse Automation: Evaluating the Perception, Planning, Control, and Grasping of Manipulation Systems

Abstract

Autonomous and reliable robotic grasping is a desirable functionality in robotic manipulation and is still an open problem. Standardized benchmarks are important tools for evaluating and comparing robotic grasping and manipulation systems among different research groups and also for sharing with the community the best practices to learn from errors. An ideal benchmarking protocol should encompass the different aspects underpinning grasp execution, including the mechatronic design of grippers, planning, perception, and control to give information on each aspect and the overall problem. This article gives an overview of the benchmarks, datasets, and competitions that have been proposed and adopted in the last few years and presents a novel benchmark with protocols for different tasks that evaluate both the single components of the system and the system as a whole, introducing an evaluation metric that allows for a fair comparison in highly cluttered scenes taking into account the difficulty of the clutter. A website dedicated to the benchmark containing information on the different tasks, maintaining the leaderboards, and serving as a contact point for the community is also provided.

CEPB dataset: a photorealistic dataset to foster the research on bin picking in cluttered environments

“…It provides three different views of the same scene seen under three lighting conditions and seven High Dynamic Range Imaging (HDRI) maps for domain randomization purposes. It is designed to be the companion dataset of the recent benchmark called Cluttered Environment Picking Benchmark (CEPB) ( DAvella et al, 2023b ), which presents a set of objects selected from existing datasets with the purpose of posing difficulties to the end-effector grasping capabilities and the perception system. Therefore, the dataset is generated synthetically using 3D CAD models of household and industrial objects coming from affirmed benchmarks like YCB ( Çalli et al, 2015 ), ACRV-APC ( Leitner et al, 2017 ; Correll et al, 2018 ), and TLESS ( Hodan et al, 2017 ) to not reinvent the wheel, and to employ existing objects used in standard manipulation benchmarks.…”

“…This section provides some background information to give a proper context for the proposed dataset, introducing the Cluttered Environment Picking Benchmark (CEPB) ( DAvella et al, 2023b ) that explains some of the design choices behind the dataset generation and structure, the Flex simulation engine used as low-level during the generation process, and the key properties of the DOPE algorithm used as baseline.…”