Policy Representation Learning for Multiobjective Reservoir Policy Design With Different Objective Dynamics

Abstract

Most water reservoir operators make use of forecasts to inform their decisions and enhance water systems flexibility and resilience by anticipating hydrological extremes. Yet, despite numerous candidate hydro‐meteorological variables and forecast horizons may potentially be beneficial to operations, the best information set for a given problem is often not evident. Additionally, in multipurpose systems characterized by multiple demands with varying vulnerabilities and temporal scales, this information set might change according to the objective tradeoff. In this work, we contribute a novel method to learn the optimal policy representation (i.e., policy input set) by combining a feature selection routine with a multiobjective Direct Policy Search framework in order to retrieve the best policy input set online (i.e., while learning the policy) and dynamically with the objective trade‐off. The selected policy search routine is the Neuro‐Evolutionary Multi‐Objective Direct Policy Search (NEMODPS) which generates flexible policy shapes adaptive to online changes in the input set. This approach is demonstrated on the case study of Lake Como (Italy), where the operating objectives are highly heterogeneous in their dynamics (fast and slow) and vulnerabilities (wet and dry extremes). We show how varying objectives, and tradeoffs therein, benefit from a different policy representation, ultimately yielding remarkable results in terms of conflict mitigation between different users. More informed policies, moreover, show higher robustness when re‐evaluated across a suite of different hydrological conditions.

A Typology for Characterizing Human Action in MultiSector Dynamics Models

“…In the former descriptive mode, AI/ML methods could be deployed alongside big social data (Lazer et al., 2009), for the realistic representation of human actors in multisector systems such as mimicking mobility patterns through a city (Moro et al., 2021) or to infer real‐world management practices (Ekblad & Herman, 2021). In prescriptive form, modeled actors could be simulated using AI/ML techniques as state‐aware agents that selectively and dynamically react to system states via reinforcement learning (e.g., see model free policy approximation methods in Powell, 2019; Bertsekas, 2019; and food‐energy‐water examples in Giuliani et al., 2021; Zaniolo et al., 2021; Cohen & Herman, 2021). In each of these endeavors, the typology can be used to properly orient and communicate the relationship between AI/ML methods and the modeled representation of human systems.…”

A Typology for Characterizing Human Action in MultiSector Dynamics Models

“…In the former descriptive mode, AI/ ML methods could be deployed alongside big social data (Lazer et al, 2009), for the realistic representation of human actors in multisector systems such as mimicking mobility patterns through a city (Moro et al, 2021) or to infer real-world management practices (Ekblad & Herman, 2021). In prescriptive form, modeled actors could be simulated using AI/ML techniques as state-aware agents that selectively and dynamically react to system states via reinforcement learning (e.g., see model free policy approximation methods in Powell, 2019;Bertsekas, 2019;and food-energy-water examples in Giuliani et al, 2021;Zaniolo et al, 2021;Cohen & Herman, 2021). In each of these endeavors, the typology can be used to properly orient and communicate the relationship between AI/ML methods and the modeled representation of human systems.…”

Multi-scale planning model for robust urban drought response

“…We formulate this cost-optimal urban water portfolio planning as a control problem [12] with flexible policy structure [37,51,52]. Figure 2(a) illustrates the method used to generate optimal planning policies π * comprising the water augmentation decisions that minimize costs while achieving no unmet demand across the streamflow ensemble in each climate scenario.…”