A Multiarmed Bandit Approach to Adaptive Water Quality Management

Abstract

Nonpoint source water quality management is challenged with allocating uncertain management actions and monitoring their performance in the absence of state-dependent decision making. This adaptive management context can be expressed as a multiarmed bandit problem. Multiarmed bandit strategies attempt to balance the exploitation of actions that appear to maximize performance with the exploration of uncertain, but potentially better, actions. We performed a test of multiarmed bandit strategies to inform adaptive water quality management in Massachusetts, USA. Conservation and restoration practitioners were tasked with allocating household wastewater treatments to minimize N inputs to impaired waters. We obtained time series of N monitoring data from 3 wastewater treatment types and organized them chronologically and randomly. The chronological data set represented nonstationary performance based on recent monitoring data, whereas the random data set represented stationary performance. We tested 2 multiarmed bandit strategies in hypothetical experiments to sample from the treatment data through 20 sequential decisions. A deterministic probability-matching strategy allocated treatments with the highest probability of success regarding their performance at each decision. A randomized probabilitymatching strategy randomly allocated treatments according to their probability of success at each decision. The strategies were compared with a nonadaptive strategy that equally allocated treatments at each decision. Results indicated that equal allocation is useful for learning in nonstationary situations but tended to overexplore inferior treatments and thus did not maximize performance when compared with the other strategies. Deterministic probability matching maximized performance in many stationary situations, but the strategy did not adequately explore treatments and converged on inferior treatments in nonstationary situations. Randomized probability matching balanced performance and learning in stationary situations, but the strategy could converge on inferior treatments in nonstationary situations. These findings provide evidence that probability-matching strategies are useful for adaptive management. Integr Environ Assess Manag 2020;16: 841-852.

Trends of surface water quality of the Krishna River, India during the urbanization process

“…Scrutiny of various associated methodologies, applications, and considerations of policies for water quality management may be carried (Dreizis, 2020). Martin and Johnson (2020) presented a multiarmed bandit approach for adaptive water quality management. Salehi et al (2020) applied principal component analysis (PCA) to correlate parameters affecting water quality, their origins, and seasonal variations.…”

“…Scrutiny of various associated methodologies, applications, and considerations of policies for water quality management may be carried (Dreizis, 2020). Martin and Johnson (2020) presented a multiarmed bandit approach for adaptive water quality management. Salehi et al.…”

Detection of District Heating Pipe Network Leakage Fault Using UCB Arm Selection Method

“…Reinforcement learning is a powerful unsupervised learning method in which the environment gives agent feedback and the agent selects the optimal action with the goal of obtaining the maximum expected cumulative reward [15]. Based on the idea of "only using the current state to obtain the optimal action" in reinforcement learning, this paper proposes a method for the rapid online detection of pipe network leakage faults based on Contextual Bandit [16,17]. In this paper, reinforcement learning is used to carry out some exploratory research in the field of pipe leakage fault detection.…”