A Parallel Deep Reinforcement Learning Framework for Controlling Industrial Assembly Lines

Abstract

Decision-making in a complex, dynamic, interconnected, and data-intensive industrial environment can be improved with the assistance of machine-learning techniques. In this work, a complex instance of industrial assembly line control is formalized and a parallel deep reinforcement learning approach is presented. We consider an assembly line control problem in which a set of tasks (e.g., vehicle assembly tasks) needs to be planned and controlled during their execution, with the aim of optimizing given key performance criteria. Specifically, the aim will be that of planning the task in order to minimize the total time taken to execute all the tasks (also called cycle time). Tasks run on workstations in the assembly line. To run, tasks need specific resources. Therefore, the tackled problem is that of optimally mapping tasks and resources to workstations, and deciding the optimal execution times of the tasks. In doing so, several constraints need to be respected (e.g., precedence constraints among the tasks, constraints on needed resources to run tasks, deadlines, etc.). The proposed approach uses deep reinforcement learning to learn a tasks/resources mapping policy that is effective in minimizing the resulting cycle time. The proposed method allows us to explicitly take into account all the constraints, and, once training is complete, can be used in real time to dynamically control the execution of tasks. Another motivation for the proposed work is in the ability of the used method to also work in complex scenarios, and in the presence of uncertainties. As a matter of fact, the use of deep neural networks allows for learning the model of the assembly line problem, in contrast with, e.g., optimization-based techniques, which require explicitly writing all the equations of the model of the problem. In order to speed up the training phase, we adopt a learning scheme in which more agents are trained in parallel. Simulations show that the proposed method can provide effective real-time decision support to industrial operators for scheduling and rescheduling activities, achieving the goal of minimizing the total tasks’ execution time.

Interpretable Input-Output Hidden Markov Model-Based Deep Reinforcement Learning for the Predictive Maintenance of Turbofan Engines

“…The trial-and-error learning through the interaction with the environment and not requiring pre-collected data and prior expert knowledge allows RL algorithms to adapt to uncertain conditions, which is also discussed by Panzer and Bender (2022). Some applications can be found in manufacturing, for instance, in scheduling tasks as an example demonstrated by Dong, Xue, Xiao and Li (2020), maintenance as a case study researched by Rodríguez, Kubler, de Giorgio, Cordy, Robert and Le Traon (2022); Yousefi, Tsianikas and Coit (2022), process control described by the authors Spielberg, Tulsyan, Lawrence, Loewen and Gopaluni (2020), energy management example elaborated by Lu, Li, Li, Jiang and Ding (2020), assembly task mentioned by Tortorelli, Imran, Delli Priscoli and Liberati (2022), and robot manipulation that in detail has been discussed by Beltran-Hernandez, Petit, Ramirez-Alpizar and Harada (2020); Schoettler, Nair, Luo, Bahl, Ojea, Solowjow and Levine (2020).…”

Advances in Machine Condition Monitoring and Fault Diagnosis

“…Finally, the optimal control of industrial assembly lines was studied in Reference [10]. In this work, a complex instance of industrial assembly line control is formalized and a parallel deep reinforcement learning approach is presented.…”

“…Finally, the functions of condition monitoring and fault diagnosis are further expanded in recent years. It is no longer limited to assisting in the operation and maintenance of equipment, but extends to the optimization of complex workflows, and even directly participates in the operation of machines, providing real-time and effective guidance for their control operations [9,10], thus greatly improving the intelligence and efficiency of machine operations. However, the premise of this is that the results of condition monitoring and fault diagnosis must be correct and reliable.…”