Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set

Abstract

Abstract-In this paper we present the Yale-CMU-Berkeley (YCB) Object and Model set, intended to be used to facilitate benchmarking in robotic manipulation, prosthetic design and rehabilitation research. The objects in the set are designed to cover a wide range of aspects of the manipulation problem; it includes objects of daily life with different shapes, sizes, textures, weight and rigidity, as well as some widely used manipulation tests. The associated database provides highresolution RGBD scans, physical properties, and geometric models of the objects for easy incorporation into manipulation and planning software platforms. In addition to describing the objects and models in the set along with how they were chosen and derived, we provide a framework and a number of example task protocols, laying out how the set can be used to quantitatively evaluate a range of manipulation approaches including planning, learning, mechanical design, control, and many others. A comprehensive literature survey on existing benchmarks and object datasets is also presented and their scope and limitations are discussed. The set will be freely distributed to research groups worldwide at a series of tutorials at robotics conferences, and will be otherwise available at a reasonable purchase cost. It is our hope that the ready availability of this set along with the ground laid in terms of protocol templates will enable the community of manipulation researchers to more easily compare approaches as well as continually evolve benchmarking tests as the field matures.

Push-Net: Deep Planar Pushing for Objects with Unknown Physical Properties

“…Real robotic experiments demonstrate the capability of Push-Net to push real objects with unknown physical properties and verify the necessity of push history for robust pushing. Test objects include a subset of YCB dataset [3] and common household objects.…”

Grasping bulky objects with two anthropomorphic hands

“…To illustrate the performance of the proposed approach, we used 6 object models from the GrabCAD repository [40], which offers a huge variety of objects of different sizes, and 4 objects from the YCB object data base [41], that have been obtained using a high-resolution scan. It was assumed that each object was made of hard plastic or wood, thus the friction coefficient was established as 0.4 (the range of mechanizable plastics is between 0.08 and 0.6).…”

Benchmarking In-Hand Manipulation

“…We defined several tasks for many of the YCB objects [8]. These tasks contain initial and desired contact regions C i , C d , and are available in our protocol's associated website.…”

“…While some benchmark examples are available for evaluating grasp planning in a simulated environment [6], [7], we believe that for contact-rich tasks such as grasping and inhand manipulation it is of fundamental importance to evaluate the performance of the system in a physical, real-world scenario. Therefore, we propose a series of tasks to evaluate the capabilities of a robotic system to plan and execute inhand manipulation using a set of objects available from the Yale-CMU-Berkeley (YCB) object and model set [8]. This set provides both object meshes and physical objects, so that different research groups can rely on the same test set.…”