Assessing Adaptive Irrigation Impacts on Water Scarcity in Nonstationary Environments—A Multi‐Agent Reinforcement Learning Approach

Abstract

One major challenge in water resource management is to balance the uncertain and nonstationary water demands and supplies caused by the changing anthropogenic and hydroclimate conditions. To address this issue, we developed a reinforcement learning agent‐based modeling (RL‐ABM) framework where agents (agriculture water users) are able to learn and adjust water demands based on their interactions with the water systems. The intelligent agents are created by a reinforcement learning algorithm adapted from the Q‐learning algorithm. We illustrated this framework in a case study where the RL‐ABM is two‐way coupled with the Colorado River Simulation System (CRSS), a long‐term planning model used for the administration of the Colorado River Basin, for assessing agriculture water uses impacts on water scarcity. Seventy‐eight intelligent agents are simulated, which can be grouped into three categories based on their parameter values: the “aggressive” (swift actions; low regrets), the “forward‐looking conservative” (mild actions; high regrets; fast learning), and the “myopic conservative” (mild actions; median regrets; slow learning). The ABM‐CRSS results showed that the major reservoirs in the Upper Colorado Basin might experience more frequent water shortages due to the increasing water uses compared to the original CRSS results. If the drought continues, the case study also demonstrates that agents can learn and adjust their demands.

Agent-Based Modeling in Water Science: From Macroscale to Microscale

“…125,165 The adaptation of decision-making processes can be strengthened through the integration of ABM and reinforcement learning, allowing for a more realistic simulation of farmers' decision-making processes by enabling agents to learn and improve their decision-making abilities to optimize long-term rewards. 113,121 ABM application to farming activities assists in the development of efficient policies that encourage farmers to adopt sustainable agricultural practices.…”

A Typology for Characterizing Human Action in MultiSector Dynamics Models

“…For example, an actor might be modeled with the capacity to switch from a utility maximization to a risk avoidance behavioral model in response to a damaging event. The axes can further be used to indicate whether actors are state-aware, the degree of this awareness, and their associated ability to learn about and adapt to the system over time such as through the selective and dynamic use of state information through reinforcement learning (Bertoni et al, 2020;Hung & Yang, 2021). 1(f) Decision Making Model: Empiricism-Distinguishes whether the behavioral model of the actor is rooted in the theory of a specific discipline (e.g., economic utility maximization) or developed in an empirical fashion relying on real-world information (observed data, surveys, etc., Janssen & Ostrom, 2006).…”

“…For example, an actor might be modeled with the capacity to switch from a utility maximization to a risk avoidance behavioral model in response to a damaging event. The axes can further be used to indicate whether actors are state‐aware, the degree of this awareness, and their associated ability to learn about and adapt to the system over time such as through the selective and dynamic use of state information through reinforcement learning (Bertoni et al., 2020; Hung & Yang, 2021).…”

Linking Critical Zone With Watershed Science: The Example of the Heihe River Basin

“…Although significant advances have been made with HRB models, they are still deficient in modeling geomorphological processes (Pan, Cai, & Geng, 2021), lateral aquatic carbon transport processes (Song & Wang, 2021), and mineral‐organic interactions in the CZ. ABMs provide flexible methodologies for representing human behaviors and their interconnections and have been successfully integrated within the watershed system model to represent human activities and their interactions with the natural system (Du et al., 2020, 2022; Hung & Yang, 2021; Yuan et al., 2021). CZ science can learn from the experience of watershed science in this area to improve the understanding of the impact of human activities on CZ processes.…”