A web-based infrastructure for recording user demonstrations of mobile manipulation tasks

Abstract

Abstract-Learning from demonstration (LfD) is a common technique applied to many problems in robotics, such as populating grasp databases, training for reinforcement learning of high-level skill sets and bootstrapping motion planners. While such approaches are generally highly valued, they rely on the often time-consuming process of gathering user demonstrations, and hence it becomes difficult to attain a sizeable dataset. In this paper, we present a tool capable of recording large numbers of high-dimensional demonstrations of mobile manipulation tasks provided by non-experts in the field. Our tool accomplishes this via a web interface that requires no additional software to be installed beyond a web browser, as well as a scalable architecture that is capable of supporting 10 concurrent demonstrators on a single server. Our architecture employs a lightweight simulation environment to reduce unnecessary computations and improve performance. Furthermore, we show how our tool can be used to gather a large set of demonstrations of a mobile manipulation task by leveraging existing crowdsource platforms. The data set collected has been made available to the robotics community. We also present experiments in which we apply demonstrations collected through our infrastructure to teach a robot how to grasp, to teach a robot how to perform dexterous manipulation tasks such as scooping and to accelerate motion planning for full-body manipulation tasks.

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

“…Most related to our work, several works have leveraged IL for MM tasks [5,6,7,8]. [5] presented a web-based tool for crowdsourcing a large scale dataset of MM tasks, and used it in combination with motion planning for execution on the robot.…”

“…Most related to our work, several works have leveraged IL for MM tasks [5,6,7,8]. [5] presented a web-based tool for crowdsourcing a large scale dataset of MM tasks, and used it in combination with motion planning for execution on the robot. [6] and [7] collected RGBD observations of humans performing tasks such as door opening and tabletop object manipulation, and used hypergraph optimization and a search procedure respectively to adapt these trajectories to be executable by a robot.…”

“…Unfortunately, the vast state space in MM has inhibited data collection from humans due to the difficulty of annotating substantial portions of the state space with good actions. Prior work has avoided this issue by instrumenting the environment to ease the state coverage burden [5,6,7,8]. However, even if there were a way to collect full human demonstrations without instrumentation, learning from such datasets would also pose a challenge due to covariate shift [9], where trained policies visit states not covered by the demonstrations -an outcome that is more likely in MM.…”

Organizing objects by predicting user preferences through collaborative filtering

“…For example, Sorokin et al (2010) utilized crowdsourcing to teach robots how to grasp new objects. Moreover, several researchers have used crowdsourcing to facilitate learning manipulation tasks from large numbers of human demonstrations (Chung et al, 2014; Ratner et al, 2015; Toris et al, 2014, 2015). In the context of learning user preferences, Jain et al (2015) recently presented a new crowdsourcing platform where non-experts can label segments in robot trajectories as desirable or not.…”

Collaborative Filtering for Predicting User Preferences for Organizing Objects

“…For example, Sorokin et al utilized crowdsourcing to teach robots how to grasp new objects [47]. Moreover, several researchers have used crowdsourcing to facilitate learning manipulation tasks from large numbers of human demonstrations [9,39,50,51]. In the context of learning user preferences, Jain et al recently presented a new crowdsourcing platform where nonexperts can label segments in robot trajectories as desirable or not [18].…”