Semantic State Estimation in Cloth Manipulation Tasks

Abstract

Understanding of deformable object manipulations such as textiles is a challenge due to the complexity and high dimensionality of the problem. Particularly, the lack of a generic representation of semantic states (e.g., crumpled, diagonally folded) during a continuous manipulation process introduces an obstacle to identify the manipulation type. In this paper, we aim to solve the problem of semantic state estimation in cloth manipulation tasks. For this purpose, we introduce a new large-scale fully-annotated RGB image dataset showing various human demonstrations of different complicated cloth manipulations. We provide a set of baseline deep networks and benchmark them on the problem of semantic state estimation using our proposed dataset. Furthermore, we investigate the scalability of our semantic state estimation framework in robot monitoring tasks of long and complex cloth manipulations. We also release the dataset and source code of the baseline models: https://github.com/BOBaraki/ state-estimation-for-continuous-cloth-manipulation

Garment Manipulation Dataset for Robot Learning by Demonstration Through a Virtual Reality Framework

“…Those works are mainly centred in studying the actions rather than the states of the piece of fabric, and for that reason, may not be as useful when trying to understand the evolution of garments between folding sates. Others use RGB-D (or RGB) images to perceive the distribution of the garment [1][2][3][4][5][6][7][8]. These approaches have to estimate the occluded parts of the piece of fabric and, for that reason, might not be as helpful when high precision methods are required.…”

“…One of the main challenges when trying to develop a garment-based dataset is whether to use real pieces of fabric or to use simulated ones. Currently, most of the available datasets are based on RGB-D images coming from real clothing data [1][2][3][4][5][6][7][8]. Despite the convenience of having real data, it is very hard to extract the ground truth information from garments and humans during a manipulation sequence.…”