Distributional Reinforcement Learning for Scheduling of Chemical Production Processes

Abstract

Reinforcement Learning (RL) has recently received significant attention from the process systems engineering and control communities. Recent works have investigated the application of RL to identify optimal scheduling decision in the presence of uncertainty. In this work, we present a RL methodology tailored to efficiently address production scheduling problems in the presence of uncertainty. We consider commonly imposed restrictions on these problems such as precedence and disjunctive constraints which are not naturally considered by RL in other contexts. Additionally, this work naturally enables the optimization of risk-sensitive formulations such as the conditional value-at-risk (CVaR), which are essential in realistic scheduling processes. The proposed strategy is investigated thoroughly in a parallel batch production environment, and benchmarked against mixed integer linear programming (MILP) strategies. We show that the policy identified by our approach is able to account for plant uncertainties in online decision-making, with expected performance comparable to existing MILP methods. Additionally, the framework gains the benefits of optimizing for risk-sensitive measures, and identifies online decisions orders of magnitude faster than the most efficient optimization approaches. This promises to mitigate practical issues and ease in handling realizations of process uncertainty in the paradigm of online production scheduling.

Industrial data science – a review of machine learning applications for chemical and process industries

“…Fig. 37 and 38 show one idea proposed in recent work 221 and a corresponding schedule generated for the case study detailed there.…”

Integrating Continuous and Batch Processes with Shared Resources in Closed-Loop Scheduling: A Case Study on Tuna Cannery

“…However, some well-known drawbacks limit the use of such AI approaches in complex industrial scheduling: (a) guaranteeing state constraints is difficult; (b) the scale is limited to small/medium size problems, otherwise the action-state space is extremely large to approximate a value function fairly enough; (c) learning optimal policies under uncertainty requires having a detailed simulation model to create extensive virtual data from thousands of runs with different state-input situations; (d) any change in the actual system structure (e.g., a new constraint or product) requires generating new valid data and re-training the AI model. In fact, the authors of a related work report that mixed-integer linear programming (MILP) outperforms reinforcement learning (RL) in terms of optimality, despite having done policy training with about 450,000 simulations of the uncertain model. Note also that such an optimality gap was already reported in a problem of about a thousand decision variables and the industrial case study tackled in this paper has more than five thousand, as will be shown in the next sections.…”