Skill-based Multi-objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration

Abstract

In modern industrial settings with small batch sizes it should be easy to set up a robot system for a new task. Strategies exist, e.g. the use of skills, but when it comes to handling forces and torques, these systems often fall short. We introduce an approach that provides a combination of task-level planning with targeted learning of scenario-specific parameters for skill-based systems. We propose the following pipeline: the user provides a task goal in the planning language PDDL, then a plan (i.e., a sequence of skills) is generated and the learnable parameters of the skills are automatically identified, and, finally, an operator chooses reward functions and hyperparameters for the learning process. Two aspects of our methodology are critical: (a) learning is tightly integrated with a knowledge framework to support symbolic planning and to provide priors for learning, (b) using multi-objective optimization. This can help to balance key performance indicators (KPIs) such as safety and task performance since they can often affect each other. We adopt a multi-objective Bayesian optimization approach and learn entirely in simulation. We demonstrate the efficacy and versatility of our approach by learning skill parameters for two different contact-rich tasks. We show their successful execution on a real 7-DOF KUKA-iiwa manipulator and outperform the manual parameterization by human robot operators.

Learning Skill-based Industrial Robot Tasks with User Priors

“…In contrast to the previous policy types, this type of policy is harder to interpret, and it is generally a difficult task to know why the robot is choosing a specific reaction to an environmental change. Therefore, [3] and [2] suggest to learn interpretable policies based on BTs and a MG [4] that are well suited for the requirements of an industrial environment.…”

“…It provides a world model (digital twin) and a skill representation based on behavior trees (BT), and has an integrated task planner. 2) An RL framework that integrates optimizers and provides a simulation as well as reward calculation [3], [2].…”

“…The operator can state lower and upper bounds for the parameters to be learned. A more detailed description can be found in [2]. In this work, we additionally allow for specification of a unimodal or multimodal prior for the optimum.…”

“…Robot skills (or just skills) are semantically defined parametric actions where parameters have to be chosen based on the task at hand through planning, sensing and knowledge integration. For contact-rich tasks, however, it can still be very challenging to find well-functioning skill parameter values, as even human operators may encounter difficulties identifying a successful and robust parameter set [2]. One solution is to allow a robot system to find these parameters through reinforcement learning (RL) directly on the task.…”

“…(1) Learning often needs to balance various key performance indicators (KPIs) such as robot speed, safety or the need to minimize interaction forces with manufacturing parts. While [3] is able to handle only single-objective learning, we argue that many tasks are best described as multiobjective learning problems where the outcome is a variety of policies for different trade-offs between the objectives [2]. An operator can then choose a solution with the desired properties.…”