A novel nested stochastic dynamic programming (nSDP) and nested reinforcement learning (nRL) algorithm for multipurpose reservoir optimization

Abstract

In this article we present two novel multipurpose reservoir optimization algorithms named nested stochastic dynamic programming (nSDP) and nested reinforcement learning (nRL). Both algorithms are built as a combination of two algorithms; in the nSDP case it is (1) stochastic dynamic programming (SDP) and (2) nested optimal allocation algorithm (nOAA) and in the nRL case it is (1) reinforcement learning (RL) and (2) nOAA. The nOAA is implemented with linear and non-linear optimization. The main novel idea is to include a nOAA at each SDP and RL state transition, that decreases starting problem dimension and alleviates curse of dimensionality. Both nSDP and nRL can solve multi-objective optimization problems without significant computational expenses and algorithm complexity and can handle dense and irregular variable discretization. The two algorithms were coded in Java as a prototype application and on the Knezevo reservoir, located in the Republic of Macedonia. The nSDP and nRL optimal reservoir policies were compared with nested dynamic programming policies, and overall conclusion is that nRL is more powerful, but significantly more complex than nSDP.

Multiobjective Direct Policy Search Using Physically Based Operating Rules in Multireservoir Systems

“…The analysis of the reservoir network resulted in a schematization of the system, which was the base for the operational model (Fig. 3 (Frank et al, 2016) with the M5 Rules classifier. For more information on the data-driven model and its accuracy, see Appendix S1.…”

“…The daily inflow time series are freely available for 2002-2015 from XM Compañía de Expertos en Mercados (2017), except INPorce3, which is available from 2011, the year when Porce III was put into operation. Therefore, INPorce3 has been extended to 2002-2015 with a validated data-driven model of high accuracy (RMSE 18.4 m 3 /s, correlation coefficient 0.9869) that has been created using the Weka Workbench software (Frank et al, 2016) with the M5 Rules classifier. For more information on the data-driven model and its accuracy, see Appendix S1.…”

“…Around 30-40 years ago, multiobjective problems were approached by means of multicriteria decision analysis (MCDA); for a review of the most common MCDA methods, see Chankong and Haimes (1983). Multicriteria decision analysis methods assume a priority ranking between objectives based on "a priori" (i.e., prior to optimization) preferences of a decision maker (DM), defining weights that are applied to the individual objectives and then aggregating them; see Delipetrev et al (2017) and Bai et al (2017) for recent applications. The multiobjective problem is thus converted into a singleobjective problem that (through the application of any single-objective optimization technique) allows for identification of one optimal compromise solution.…”

Reinforcement learning-based real-time control of coastal urban stormwater systems to mitigate flooding and improve water quality

“…Classical tabular RL is closely related to dynamic programming and has been explored for multiobjective reservoir management. [22][23][24][25][26] However, because tabular RL is limited to systems with relatively small numbers of possible states and actions, deep reinforcement learning (also widely referred to as RL), which uses neural networks as function approximators instead of using lookup tables, has been used for control of more complex systems. 27,28 This approach to learning allows RL increased flexibility to optimize control actions, balance competing objectives based on the formulation of the reward function, and has the potential to continually adapt system controls to evolving environmental conditions (e.g., increased runoff from urbanization or climate change).…”

Using Cellular Automata Approach to Optimize the Hydropower Reservoir Operation of Folsom Dam

“…Different methods have been examined to evaluate the effectiveness of re-operating or optimizing the operation of reservoirs in order to maximize the benefits, especially hydropower generation, in particular using optimization methods e.g., [15][16][17][18]. Researchers have tested classic optimization approaches such as linear programming (LP) [19,20], non-linear programming (NLP) [21,22], and dynamic programming (DP) [23,24] to solve reservoir operation problems in different forms. However, these studies also simplified objective functions and constraints in a linear model, which makes them unsuitable for optimizing hydropower generation [25].…”