Dynamic multi-objective optimisation using deep reinforcement learning: benchmark, algorithm and an application to identify vulnerable zones based on water quality

Abstract

Dynamic multi-objective optimisation problem (DMOP) has brought a great challenge to the reinforcement learning (RL) research area due to its dynamic nature such as objective functions, constraints and problem parameters that may change over time. This study aims to identify the lacking in the existing benchmarks for multi-objective optimisation for the dynamic environment in the RL settings. Hence, a dynamic multi-objective testbed has been created which is a modified version of the conventional deep-sea treasure (DST) hunt testbed. This modified testbed fulfils the changing aspects of the dynamic environment in terms of the characteristics where the changes occur based on time. To the authors' knowledge, this is the first dynamic multi-objective testbed for RL research, especially for deep reinforcement learning. In addition to that, a generic algorithm is proposed to solve the multi-objective optimisation problem in a dynamic constrained environment that maintains equilibrium by mapping different objectives simultaneously to provide the most compromised solution that closed to the true Pareto front (PF). As a proof of concept, the developed algorithm has been implemented to build an expert system for a real-world scenario using Markov decision process to identify the vulnerable zones based on water quality resilience in São Paulo, Brazil. The outcome of the implementation reveals that the proposed parity-Q deep Q network (PQDQN) algorithm is an efficient way to optimise the decision in a dynamic environment. Moreover, the result shows PQDQN algorithm performs better compared to the other state-of-the-art solutions both in the simulated and the real-world scenario.

Deep reinforcement learning for automated search of model parameters: photo-fenton wastewater disinfection case study

“…This approach to optimization has also been applied to real-world problems and, in particular, to chemical industry, for example in real-time optimization of hydrocracking [19], in batch bio-process optimization for finding alternatives to fossil based materials [20], in batch optimization of bioreactors for food industry [21], in real-time detection of pollution risk due to wastewater [22] and in the analysis of material qualities like hardness of aluminum alloys [23]. It has also been applied in other domains such as health care, for melanoma's gene regulation [24] or protein folding problems in the fight against hereditary diseases [25], or in the field of energy, as in [26] to manage the electric power in a building or a small city, or in [27] to maximize electrical energy generation with acceptable emission levels.…”

A novel machine learning application: Water quality resilience prediction Model

“…Several metrics/measures have been developed and introduced in literature to quantify resilience for different contexts in different water systems (Hashimoto et al, 1982;Fowler et al, 2003;Matrosov et al, 2012;Jung, 2013;Paton et al, 2014;Butler et al, 2016). This study inspired by the 'volume-based resilience metrics' method, introduced in the study by Roach et al (2018), and formulated by Hasan et al (2019) and McClymont et al (2020), to encapsulate a suitable resilience-based performance metric for WQ evaluation. Details about formulation of WQR can be found in section 3 in SI.…”

“…This model can be used to assess water resource adaptive capacity in order to identify the most vulnerable/critical contaminated UGRHIs (so-called zones in this paper) without the need for a great deal of costly data collection and time-consuming system modelling. Additionally, it can enable water and environment sectors to map the spatial and temporal dynamics of resilience fluctuations across the study area and time (Hasan et al, 2019), without direct characterisation of the influential factors (e.g., land use pattern, hydrological parameters, etc.). The proposed predictive model can be utilised to make predictions of future WQ resilience and therefore, enables proactive resilience planning.…”

Development of local and global wastewater biochemical oxygen demand real-time prediction models using supervised machine learning algorithms

“…Due to that, operators with adequate expert knowledge are needed to be able to run such a plant. In water quality control applications, machine-learning (ML) models can be utilized (Hasan et al, 2019).…”