Systematic Performance Evaluation of Reinforcement Learning Algorithms Applied to Wastewater Treatment Control Optimization

Abstract

Treatment of wastewater using activated sludge relies on several complex, nonlinear processes. While activated sludge systems can provide high levels of treatment, including nutrient removal, operating these systems is often challenging and energy intensive. Significant research investment has been made in recent years into improving control optimization of such systems, through both domain knowledge and, more recently, machine learning. This study leverages a novel interface between a common process modeling software and a Python reinforcement learning environment to evaluate four common reinforcement learning algorithms for their ability to minimize treatment energy use while maintaining effluent compliance within the Benchmark Simulation Model No. 1 (BSM1) simulation. Three of the algorithms tested, deep Q-learning, proximal policy optimization, and synchronous advantage actor critic, generally performed poorly over the scenarios tested in this study. In contrast, the twin delayed deep deterministic policy gradient (TD3) algorithm consistently produced a high level of control optimization while maintaining the treatment requirements. Under the best selection of state observation features, TD3 control optimization reduced aeration and pumping energy requirements by 14.3% compared to the BSM1 benchmark control, outperforming the advanced domain-based control strategy of ammonia-based aeration control, although future work is necessary to improve robustness of RL implementation.

Multiobjective Optimization of Papermaking Wastewater Treatment Processes under Economic, Energy, and Environmental Goals

“…It is limited to not only the PWTP but also the general wastewater treatment operations since the dynamic optimization of the real-world application involves multiple objectives and constraints in common, and the dynamic optimization environment requires strictly high flexibility, efficiency, and robustness of the optimization system. The multiagent DRL takes into account all the features above that, also as supported by the related studies, , can contribute to the treatment process to achieve the synergistic effect of pollution elimination and carbon reduction.…”

“…Mohammadi et al used DRL to adapt to the process’s complex dynamics, optimized the WWTP for phosphorus removal control, and suggested a comparative investigation of different DRLs in an imperfect environment. Croll et al have evaluated four common DRL algorithms systematically to minimize treatment energy in wastewater treatment control . Several review papers indicated that the links of DRL study to minimize carbon emission and balance the conflicting objectives of WWTP performances are imperative and could be promising future directions …”

Deep Learning-Based Multiobjective Optimization for Balancing Effluent Quality, Operational Cost, and Greenhouse Gas Emissions in Wastewater Treatment Plant Control

“…Numerous efforts have been made to improve the control and operation of WWTPs toward low-carbon and sustainable. , It has been proven that implementing a proper operational control scheme may be effective for reducing GHG emissions in the WWTP . An improved cascade control structure could reduce the N 2 O emissions by nearly 30% with compromised cost and satisfied water quality .…”