Optimizing Long‐term Hydro‐power Production Using Markov Decision Processes

Abstract

Modelling the long‐term operation of hydroelectric systems is one of the classic applications of Markov decision process (MDP). The computation of optimal policies, for MDP models, is usually done by dynamic programming (DP) on a discretized state space. A major difficulty arises when optimizing multi‐reservoir systems, because the computational complexity of DP increases exponentially with the number of sites. This so‐called ‘curse of dimensionality’ has so far restricted the applicability of DP to very small systems (2 or 3 sites). Practitioners have thus had to resort to other methodologies for the long‐term planning, often at the expense of rigour, and without reliable error estimates. This paper surveys recent research of MDP computation, with application to hydro‐power systems. Three main approaches are discussed: (i) discrete DP, (ii) numerical approximation of the expected future reward function, and (iii) analytic solution of the DP recursion.

Managing a reservoir-based hydro-energy plant: building understanding in the buy and sell decisions in a changing environment

“…For instance, Lamond and Boukhtouta (1996) provide a survey of the use of Markov decision processes to achieve long-term optimisation of hydro-power production. Lyra et al (1996) address the issues of conflicting interests and provide a survey of quantitative tools (multi-criteria optimisation) designed to assist in settling such disputes.…”

Une Étude Numérique De La Discrétisation Des Apports Aléatoires Pour Un Réservoir Non Saisonnier

“…ou ni et 722 sont donnes par (9). La table 4 donne le nombre de points oti la loi est non nuUe, pour chacune de nos experiences numeriques, en fonction du pas de discretisation k.…”

“…Toutefois, les equations obtenues necessitent habituellement l'utilisation de methodes d'optimisation numeriques, et les modeles a reservoirs multiples demeurent extremement difHciles a resoudre numeriquement a cause de Texplosion dimensionnelle. Voir, e.g., [5] pour une introduction aux methodes de planification de la production d'un systeme hydroelectrique, et [9,16,17] pour une revue de la litterature sur l'utilisation de la programmation dynamique en gestion des reservoirs.…”

Dynamic water resources planning with locational release and annual consumption constraints

“…Various methods were developed to deal with this challenge. Approximation for value functions, such as piecewise linear function or neural network, are investigated in [7] and [8]. Approximation in policy space, such as various operation rules and heuristics, are studied in [5] and [6].…”

“…Markov Decision Process (MDP) [13]- [15] is widely used to formulate the dynamic water resources planning due to its ability to cope with nonlinear and stochastic characteristics of such problems [2] [8]. However, it faces the well known challenge of huge decision or policy space, often referred to as "curse of dimensionality" as surveyed in [8].…”