Simulation and deep reinforcement learning for adaptive dispatching in semiconductor manufacturing systems

Abstract

Fabrication areas in semiconductor industry are considered one of the most complex production systems. This complexity is caused by the high-mix of products and end-user market-based demands in that industry. Its dynamic and challenging processing requirements affect the handling capabilities of traditional production management paradigms. In this paper, we propose an application for dispatching and resources allocation through reinforcement learning. The application is based on a discrete-event simulation model for a case study of a real semiconductor manufacturing system. The model is built using both data-driven and agent-based approaches. The model simulates the various processing aspects that are present normally in these complex systems. The model's agents are responsible for dispatching tasks and allocation of the different system's resources. They employ Deep-Q-Network reinforcement learning. They learn simultaneously through the model execution. An independent Deep-Q-Network is trained for each agent. The model provides the training environment for the agents in which their decisions are applied and assessed for their adequacy. Our formulation of the environment's state and the reward function for the learning algorithms creates cooperative decision-making policies for the agents. This results in improving the global performance of the whole system, and the performance of each agent's resources. Our approach is compared to heuristics-based strategies that are applied in our case study. It achieved better production performance than the currently applied strategy.

Problems and Solution Methods of Machine Scheduling in Semiconductor Manufacturing Operations: A Survey

“…For instance, adaptive scheduling solutions that automatically choose the most suitable dispatching rules based on system status have been used in [60,197]. The AI-based technologies, e.g., DQL and case-based reasoning, could also be leveraged to train the system and reduce computational time [191][192][193].…”

“…Within the method, an OCBA procedure is used to reduce simulation samples and a reinforcement learning algorithm is used to dynamically adjust the parameters of the metaheuristic [190]. Other AI-based methods, such as convolutional neural network and asynchronous advanced actor critic-based method (CNN-A3C) [191], case-based reasoning [192], and DQL [193], have also been utilized recently.…”

Schlably: A Python Framework for Deep Reinforcement Learning Based Scheduling Experiments

“…Yet, all empirical studies apply a very similar experimental setup consisting of the same main software components: An environment representing the production facility layout and logic, a scheduling problem generator, a DRL agent algorithm, and logging as well as evaluation tools. The difference between different experimental setups usually lies within one or more of these components, for example by incorporating a new DRL algorithm [10,11,12], interaction logic between agent and environment [6,3], training procedure [13], learning objective [14,12] or additional problem constraint [14,15]. Regardless of large overlaps, all researchers implement their own individual experimentation framework with the following two consequences: Large initial ramp-up efforts when experimenting with new methodologies or custom problem settings, and scarcity of empirical comparisons to other works.…”

Traffic Flow Speed Prediction in Overhead Transport Systems for Semiconductor Fabrication Using Dense-UNet

“…The output from the convolutional layer is typically used for the next operation via a non-linear activation function. We used linear activation, as in Equation (2), ELU [14], as in Equation (3), and sigmoid, as in Equation (4).…”

“…Efficient operation of overhead hoist transport (OHT) systems is important for the productivity of semiconductor processes [1]. In particular, it is important to predict traffic flow and congestion over time because OHT operations, such as dispatching [2][3][4] and routing [5][6][7][8], are highly dependent on traffic conditions. In this study, the p OHT congestion prediction issue is addressed based on volume data.…”