A Reinforcement Learning Model to Assess Market Power Under Auction-Based Energy Pricing

Abstract

Abstract-Auctions serve as a primary pricing mechanism in various market segments of a deregulated power industry. In day-ahead (DA) energy markets, strategies such as uniform price, discriminatory, and second-price uniform auctions result in different price settlements and thus offer different levels of market power. In this paper, we present a nonzero sum stochastic game theoretic model and a reinforcement learning (RL)-based solution framework that allow assessment of market power in DA markets. Since there are no available methods to obtain exact analytical solutions of stochastic games, an RL-based approach is utilized, which offers a computationally viable tool to obtain approximate solutions. These solutions provide effective bidding strategies for the DA market participants. The market powers associated with the bidding strategies are calculated using well-known indexes like Herfindahl-Hirschmann index and Lerner index and two new indices, quantity modulated price index (QMPI) and revenue-based market power index (RMPI), which are developed in this paper. The proposed RL-based methodology is tested on a sample network.Index Terms-Auctions, average reward stochastic games, competitive Markov decision processes (CMDPs), deregulated electricity markets, market power, reinforcement learning (RL).

Hierarchically Structured Energy Markets as Novel Smart Grid Control Approach

“…Of course, these rules have also major influence on the behavior of the agents; different bidding strategies are considered by Kian et al [8] and optimal strategies are known as well [3]. Also classical auction schemes as Vickrey-Clarke-Groves auctions [6] or other game theoretic approaches and approaches from computational learning theory have been applied [12]. It should be clear, that there may be large differences between the considered markets if one looks at their inner structure, but they all fit in the scheme described within this paper.…”

Societies of Intelligent Agents

“…Agent-based economic models can help understand market instabilities and other non-equilibrium phenomena [4]. These models sometimes include agents that learn by reinforcement from the results of their behavior [5]. This suggests equating money with reward to create a society in which agents exchange reward for goods and services.…”

Evaluating the efficiency of divestiture policy in promoting competitiveness using an analytical method and agent-based computational economics

“…(1) locational marginal pricing (LMP), (2) market splitting and (3) flow-based market coupling (Krause and Andersson, 2006). An RLbased approach is utilized (Nanduri and Das, 2007) to analyze the market power in the day-ahead energy markets operated under uniform price, discriminatory and second-price uniform auctions.…”