An MDP Model-Based Reinforcement Learning Approach for Production Station Ramp-Up Optimization: Q-Learning Analysis

Abstract

Ramp-up is a significant bottleneck for the introduction of new or adapted manufacturing systems. The effort and time required to ramp-up a system is largely dependent on the effectiveness of the human decision making process to select the most promising sequence of actions to improve the system to the required level of performance. Although existing work has identified significant factors influencing the effectiveness of rampup, little has been done to support the decision making during the process. This paper approaches ramp-up as a sequential adjustment and tuning process that aims to get a manufacturing system to a desirable performance in the fastest possible time. Production stations and machines are the key resources in a manufacturing system. They are often functionally decoupled and can be treated in the first instance as independent rampup problems. Hence, this paper focuses on developing a Markov decision process (MDP) model to formalize ramp-up of production stations and enable their formal analysis. The aim is to capture the cause-and-effect relationships between an operator's adaptation or adjustment of a station and the station's response to improve the effectiveness of the process. Reinforcement learning has been identified as a promising approach to learn from rampup experience and discover more successful decision-making policies. Batch learning in particular can perform well with little data. This paper investigates the application of a Q-batch learning algorithm combined with an MDP model of the ramp-up process. The approach has been applied to a highly automated production station where several ramp-up processes are carried out. The convergence of the Q-learning algorithm has been analyzed along with the variation of its parameters. Finally, the learned policy has been applied and compared against previous ramp-up cases.

Exploring the integration of the human as a flexibility factor in CPS enabled manufacturing environments: Methodology and results

“…A special emphasis is made here on the need for adaptive interfaces as well as the importance of utilizing notions of distributed decision support system approaches such as multi-agent systems. Doltsinis et al [20] emphasise on the need for capturing operator's actions during the ramp-up phase. Operator's hardware and software adjustments during the ramp-up phase were recorded and used as part of a reinforcement learning approach to derive generalised policies for adjustment strategies leading to faster ramp-up.…”

Decision support models for production ramp-up: a systematic literature review

“…Measuring and monitoring the performance of the production system during the ramp-up is critical, as the production process during this phase is unstable and error-prone. As a result, the system needs to be interrupted and adjusted at regular time intervals until a steady-state phase is reached (Doltsinis, Ferreira, and Lohse 2014). To identify deviations from production plans, production managers need to constantly monitor the performance of the production process during the ramp-up (Winkler, Heins, and Nyhuis 2007).…”

“…Examples include the determination of production capacities, the re-sequencing or changing of processes, the determination of lot sizes, worker training or the assignment of workers to work stations (Almgren 1999b;Scholz-Reiter et al 2007;Gopal et al 2013;Doltsinis, Ferreira, and Lohse 2014;Li et al 2014). It is clear that if companies solve these planning problems by trial and error instead of following a systematic planning approach, delays and errors may occur, which increase the time to volume and put the success of product introduction at stake (Schuh, Desoi, and Tücks 2005;Doltsinis, Ratchev, and Lohse 2013).…”

“…The most important process characteristics of the ramp-up phase can be summarised as follows (cf. Baloff 1970;Wilhelm and Sastri 1979;Terwiesch and Xu, 2004;Glock, Jaber, and Zolfaghari 2012;Niroomand, Kuzgunkaya, and Bulgak 2012;Doltsinis, Ratchev, and Lohse 2013;Doltsinis, Ferreira, and Lohse 2014;Surbier, Alpan, and Blanco 2014):…”

Reinforcement Learning endowed with safe veto policies to learn the control of Linked-Multicomponent Robotic Systems

“…The success of Q-Learning has lead to many applications, such as path planning [25,31], energy management [30], routing in vehicular ad-hoc networks [42], management of water resources [27], and production planning [10].…”