Deep Reinforcement Learning for Dynamic Flexible Job Shop Scheduling with Random Job Arrival

Abstract

The production process of a smart factory is complex and dynamic. As the core of manufacturing management, the research into the flexible job shop scheduling problem (FJSP) focuses on optimizing scheduling decisions in real time, according to the changes in the production environment. In this paper, deep reinforcement learning (DRL) is proposed to solve the dynamic FJSP (DFJSP) with random job arrival, with the goal of minimizing penalties for earliness and tardiness. A double deep Q-networks (DDQN) architecture is proposed and state features, actions and rewards are designed. A soft ε-greedy behavior policy is designed according to the scale of the problem. The experimental results show that the proposed DRL is better than other reinforcement learning (RL) algorithms, heuristics and metaheuristics in terms of solution quality and generalization. In addition, the soft ε-greedy strategy reasonably balances exploration and exploitation, thereby improving the learning efficiency of the scheduling agent. The DRL method is adaptive to the dynamic changes of the production environment in a flexible job shop, which contributes to the establishment of a flexible scheduling system with self-learning, real-time optimization and intelligent decision-making.

Hierarchical Reinforcement Learning for Multi-Objective Real-Time Flexible Scheduling in a Smart Shop Floor

“…A mathematical model of the MODFJSP with random job arrival in a smart machine tool processing workshop was established. All notations used to describe the problem are summarized in Table 2 in line with the literature [18]. For each job J i , its due date D i can be calculated as…”

“…In addition, the differences among action values for s t are often very small relative to the magnitude of Q. For example, after training with the DDQN in [18], the average gap between the Q values of the best and the second-best action across visited states is roughly 0.06, whereas the average action value across those states is about 17. Action values are frequently reordered, and actions chosen by behavior strategies are correspondingly changed, which brings small amounts of noise in the updates.…”

“…The average utilization rate of the machines is denoted by U m (t), which is calculated using Equation (11) [14,18]. At rescheduling point t, the completion time of the last operation on machine M k and the current number of completed operations of job J i are denoted by CT k (t) and OP i (t), respectively.…”

“…(5) Estimated total machine load F e (t); The methods for calculation of ET e (t) and P e (t) are the same as those for the actual earliness and tardiness rate, ET a (t), and the actual earliness and tardiness penalty, P a (t), in [18], respectively.…”

“…In the actual production environment of machine tool processing, the swift completion of products results in higher inventory pressure, whereas delays in completing the job result in financial damage [18]. The total machine load not only affects financial costs but also involves energy saving and emission reduction.…”