A Q-Cube Framework of Reinforcement Learning Algorithm for Continuous Double Auction among Microgrids

Abstract

Decision-making of microgrids in the condition of a dynamic uncertain bidding environment has always been a significant subject of interest in the context of energy markets. The emerging application of reinforcement learning algorithms in energy markets provides solutions to this problem. In this paper, we investigate the potential of applying a Q-learning algorithm into a continuous double auction mechanism. By choosing a global supply and demand relationship as states and considering both bidding price and quantity as actions, a new Q-learning architecture is proposed to better reflect personalized bidding preferences and response to real-time market conditions. The application of battery energy storage system performs an alternative form of demand response by exerting potential capacity. A Q-cube framework is designed to describe the Q-value distribution iteration. Results from a case study on 14 microgrids in Guizhou Province, China indicate that the proposed Q-cube framework is capable of making rational bidding decisions and raising the microgrids’ profits.

How to trade electricity flexibility using artificial intelligence - An integrated algorithmic framework

“…[26], [27], [28] Artificial Neural Networks [11], [29], [30], [31], [32], [33], [34] [35], [12], [36], [37], [38], [39], [40] Reinforcement Learning [41], [42], [43], [44], [45] [46], [47], [48], [49], [50], [51], [52], [10], [53] Nature-Inspired Intelligence [54] [55]…”

“…Similarly, [34] propose a polynomial-time online learning algorithm for virtual trading on the day-ahead and the real-time electricity market. [30] propose a Q-learning approach for a double auction mechanism for trading between several micro-grid operators. Hence, the focus of this approach is more on bilateral trading rather than the application of trading strategies on a specific electricity market.…”

“…In the environment of a Peer-to-Peer (P2P) market, the micro-grids' objective is to learn a strategy that maximizes the financial reward for the prosumers of the respective micro-grids. In contrast to [30], [33] propose a dynamic programming algorithm. [44] and [45] combine RL approaches with Artificial Neural Networks (ANNs).…”

A Simulation Environment for Training a Reinforcement Learning Agent Trading a Battery Storage

“…Batteries have been used in conjunction with reschedulable loads to perform the rescheduling to exploit time-of-use and real-time energy pricing schemes [34][35][36] and variable intraday electricity market prices [37]. Whereas most works are aimed at existing electricity markets, a few authors have demonstrated the benefits of RL to optimize the emerging decentralized electricity system on novel markets [38,39]. The above works involved decision-making on electricity markets, which is the most common type of RL application in this category.…”

A Double-Auction Mechanism for Distribution of Electrical Supply Capacities among Enterprises of an Industrial Park

“…En la literatura se han presentado varios modelos y diseños basados en SD para abordar diferentes situaciones en entornos de licitación dinámica de mercados de comercialización de energía en micro-redes [10,17]. En algunos trabajos, se desarrollaron algoritmos de SD aplicables a micro-redes interconectadas, por ejemplo utilizando métodos de aprendizaje reforzado (reinforcement learning) [18].…”