Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space for Robotic Assembly Tasks

Abstract

This paper introduces a discrete-continuous action space to learn insertion primitives for robotic assembly tasks. Primitive is a sequence of elementary actions with certain exit conditions, such as "pushing down the peg until contact". Since the primitive is an abstraction of robot control commands and encodes human prior knowledge, it reduces the exploration difficulty and yields better learning efficiency. In this paper, we learn robot assembly skills via primitives. Specifically, we formulate insertion primitives as parameterized actions: hybrid actions consisting of discrete primitive types and continuous primitive parameters. Compared with the previous work using a set of discretized parameters for each primitive, the agent in our method can freely choose primitive parameters from a continuous space, which is more flexible and efficient. To learn these insertion primitives, we propose Twin-Smoothed Multipass Deep Q-Network (TS-MP-DQN), an advanced version of MP-DQN with twin Q-network to reduce the Q-value over-estimation. Extensive experiments are conducted in the simulation and real world for validation. From experiment results, our approach achieves higher success rates than three baselines: MP-DQN with parameterized actions, primitives with discrete parameters, and continuous velocity control. Furthermore, learned primitives are robust to sim-to-real transfer and can generalize to challenging assembly tasks such as tight round peg-hole and complex shaped electric connectors with promising success rates. Experiment videos are available at https://msc.berkeley.edu/research/insertion-primitives.html.

Factory: Fast Contact for Robotic Assembly

“…Efforts using MuJoCo or the robosuite extension [112] include [19,22,29,37,38,48,82,88,96,97,99,108,110,111]. Simulated rigid-body assembly tasks are limited to peg-inhole insertion of large pegs with round, triangular, square, and prismatic cross-sections, lap-joint mating, and one non-convex insertion [29].…”

“…Most recent works have used model-free, off-policy RL algorithms or variants, which do not predict environment response, and update an action-value function independently of the current policy (e.g., using a replay buffer). These studies have applied Q-learning [35], deep-Q networks [110], deep deterministic policy gradients (DDPG) [5,58,56,97], softactor critic [7], probabilistic embeddings [82], and hierarchical RL [32]. These algorithms are typically chosen for sample efficiency, but are often brittle and slow/unable to converge.…”