Reinforcement learning-based particle swarm optimization for sewage treatment control

Abstract

To solve the problem of high-energy consumption in activated sludge wastewater treatment, a reinforcement learning-based particle swarm optimization (RLPSO) was proposed to optimize the control setting in the sewage process. This algorithm tries to take advantage of the valid history information to guide the behavior of particles through a reinforcement learning strategy. First, an elite network is constructed by selecting elite particles and recording their successful search behavior. Then the network is trained and evaluated to effectively predict the particle velocity. In the periodic wastewater treatment process, the RLPSO runs repeatedly according to the optimized cycle. Finally, RLPSO was tested based on Benchmark Simulation Model 1 (BSM1) of sewage treatment, and the simulation results showed that it could effectively reduce the energy consumption on the premise of ensuring qualified water quality. Furthermore, the performance of RLPSO was analyzed using the benchmarks with higher dimension, which verifies the effectiveness of the algorithm and provides the possibility for RLPSO to be applied to a wider range of problems.

Systematic Performance Evaluation of Reinforcement Learning Algorithms Applied to Wastewater Treatment Control Optimization

“…Scenarios 1−3 evaluated RL agents for control of DO, a single action, while scenarios 4−6 evaluated RL agents for simultaneous "dual action" control of DO and IMLR. Action ranges were set from 0.5−2 mg/L for DO, a range of typical values also used by Lu et al, 17 and from 0 to 90000 m 3 /day for IMLR, based on the range provided in the BSM1. 18 The features in the control scenarios were chosen for their known chemical and biological process relationships based on domain expertise.…”

“…In particular, reinforcement learning (RL) stands out for its ability to perform better than human operators for certain high-dimensional, complex decision-making problems. − Unlike other ML techniques, online RL algorithms that are considered here do not learn from static historical data sets but by directly exploring different actions based on a set of observation variables, often referred to as features, to maximize a specified reward. Several recent review papers have been published evaluating various facets and implementations of RL algorithms. − While RL control optimization in the context of wastewater treatment has almost exclusively been confined to academic research, the ability of RL to generate process efficiency improvements makes it of significant interest to the industry moving forward …”

“…Therefore, we aim to determine what control strategies an RL agent would devise if given broader control of the BSM1 treatment process with the ability to control the IMLR and DO set points directly. Previous work on the BSM1 by Hernandez-Del-Olmo et al evaluated the control of DO set points by Q-learning agents, − and Lu et al evaluated the control of DO and IMLR set points by a variety of particle swarm optimization (PSO) agents . While both studies demonstrated the ability of RL control to improve operational costs, each focused on a single class of RL algorithms under a single set of input features and did not evaluate more than n = 1 agents for each scenario to demonstrate consistency of agent performance.…”

“…11−16 While RL control optimization in the context of wastewater treatment has almost exclusively been confined to academic research, the ability of RL to generate process efficiency improvements makes it of significant interest to the industry moving forward. 17 Among both domain and ML control optimization strategies, a common method for the evaluation of wastewater treatment control strategies has been the Benchmark Simulation Model No. 1 (BSM1), an anoxic−oxic treatment process simulation configured for nitrogen removal.…”

Reinforcement-learning-based parameter adaptation method for particle swarm optimization

“…Finally, the reward was determined based on whether the whole optimization problem grew. In the paper [46], the author used the Q-learning algorithm. The particle position was taken as the state input.…”