Automated synthesis of steady-state continuous processes using reinforcement learning

Abstract

Automated flowsheet synthesis is an important field in computer-aided process engineering. The present work demonstrates how reinforcement learning can be used for automated flowsheet synthesis without any heuristics or prior knowledge of conceptual design. The environment consists of a steady-state flowsheet simulator that contains all physical knowledge. An agent is trained to take discrete actions and sequentially build up flowsheets that solve a given process problem. A novel method named SynGameZero is developed to ensure good exploration schemes in the complex problem. Therein, flowsheet synthesis is modelled as a game of two competing players. The agent plays this game against itself during training and consists of an artificial neural network and a tree search for forward planning. The method is applied successfully to a reaction-distillation process in a quaternary system.

Automated Flowsheet Synthesis Using Hierarchical Reinforcement Learning: Proof of Concept

“…The present work extends the SynGameZero [22] approach by splitting the agent's actions into three hierarchically ordered levels. At first, the agent chooses a location, at second, the corresponding unit operation, and at third, a specification of the unit, if needed.…”

“…1. For a detailed description of the Syn-GameZero approach, we refer to [22] and the Supporting Information (SI). The agent's decisions during a player's turn are organized hierarchically: -Level 1: decide whether to terminate or, alternatively, select an open stream to manipulate.…”

“…The agent, which consists of several ANNs combined with a tree search, works on the different hierarchy levels. The general concept of a two-player game as presented in [22] is employed on top of this hierarchical feature. The approach is demonstrated by designing processes for ethyl tert-butyl ether (ETBE) synthesis [29][30][31].…”

“…In earlier work [22], we have taught an agent via RL to synthesize process flowsheets comprising different unit operations (reactors, distillation columns, mixers) leading to a very large decision space. To master this, flowsheet synthesis was transformed into a turn-based two-player game, which is called SynGameZero.…”

“…The approach is demonstrated by designing processes for ethyl tert-butyl ether (ETBE) synthesis [29][30][31]. This process example displays a significantly increased level of complexity compared to the problems considered in previous works [20][21][22]25], as the agent has to deal with azeotropic behavior, equilibrium reactions, and recycle streams.…”