Designing an adaptive production control system using reinforcement learning

Abstract

Modern production systems face enormous challenges due to rising customer requirements resulting in complex production systems. The operational efficiency in the competitive industry is ensured by an adequate production control system that manages all operations in order to optimize key performance indicators. Currently, control systems are mostly based on static and model-based heuristics, requiring significant human domain knowledge and, hence, do not match the dynamic environment of manufacturing companies. Data-driven reinforcement learning (RL) showed compelling results in applications such as board and computer games as well as first production applications. This paper addresses the design of RL to create an adaptive production control system by the real-world example of order dispatching in a complex job shop. As RL algorithms are "black box" approaches, they inherently prohibit a comprehensive understanding. Furthermore, the experience with advanced RL algorithms is still limited to single successful applications, which limits the transferability of results. In this paper, we examine the performance of the state, action, and reward function RL design. When analyzing the results, we identify robust RL designs. This makes RL an advantageous control system for highly dynamic and complex production systems, mainly when domain knowledge is limited.

Human-centred design in industry 4.0: case study review and opportunities for future research

“…A challenge of manufacturing today is adapting to an increasingly fluctuating environment and diverse changes to meet the demands of the market. Product life cycles are getting shorter while production batch sizes are getting smaller with dynamic product variants associated with increasing complexity, which is challenging the traditional production systems (Benabdellah et al, 2019 ; Kuhnle et al, 2021 ; Ma et al, 2017 ; Prinz et al, 2019 ; Windt et al, 2008 ; Zhu et al, 2015 ). To manage these dynamics, the industrial concept of Industry 4.0 has come about and has been accepted in both research and industry, a trend linked to digitalization and smart systems that could enable factories to achieve higher production variety with reduced downtimes while improving yield, quality, safety, and decreasing cost and energy consumption (García-Magro & Soriano-Pinar, 2019 ; Järvenpää et al, 2019 ; Napoleone et al, 2020 ; Oztemel & Gursev, 2020 ; Park & Tran, 2014 ).…”

Modular production control using deep reinforcement learning: proximal policy optimization

“…Kuhnle et al detailed that the implementation outperformed existing benchmark heuristics. In a succeeding work from Kuhnle et al (2021), they focused on state, action, and reward designs in RL production control and concluded their importance for successful learning. Even if the semiconductor example is not directly transferable to our work, inspiration about the state and action representation, as well as the reward function, and other setup parameters can be generated, especially from their 2021 publication.…”

“…For instance, if a product is loaded on an AGV, another pickup-action is not valid, just a dropdown-action is meaningful. This definition orients itself on May et al (2021) and Kuhnle et al (2021). A dropdown action is also not valid if a machine and its respective buffers are already full or have reached a fixed capacity limit of n dl being set as a parameter.…”

Deep reinforcement learning methods for structure-guided processing path optimization

“…Recently published work include the optimization of process control in sheet metal milling (Veeramani et al 2019), polymerization reaction systems (Ma et al 2019), laser welding (Günther et al 2016) and in deep drawing (Dornheim et al 2019). Operational optimization objects are amongst others material flow in industrial mining (Kumar et al 2020), preventive maintenance scheduling of flow line systems (Wang et al 2016) and job shop scheduling (Kuhnle et al 2020).…”