Application of a Gradient Descent Continuous Actor-Critic Algorithm for Double-Side Day-Ahead Electricity Market Modeling

Abstract

An important goal of China's electric power system reform is to create a double-side day-ahead wholesale electricity market in the future, where the suppliers (represented by GenCOs) and demanders (represented by DisCOs) compete simultaneously with each other in one market. Therefore, modeling and simulating the dynamic bidding process and the equilibrium in the double-side day-ahead electricity market scientifically is not only important to some developed countries, but also to China to provide a bidding decision-making tool to help GenCOs and DisCOs obtain more profits in market competition. Meanwhile, it can also provide an economic analysis tool to help government officials design the proper market mechanisms and policies. The traditional dynamic game model and table-based reinforcement learning algorithm have already been employed in the day-ahead electricity market modeling. However, those models are based on some assumptions, such as taking the probability distribution function of market clearing price (MCP) and each rival's bidding strategy as common knowledge (in dynamic game market models), and assuming the discrete state and action sets of every agent (in table-based reinforcement learning market models), which are no longer applicable in a realistic situation. In this paper, a modified reinforcement learning method, called gradient descent continuous Actor-Critic (GDCAC) algorithm was employed in the double-side day-ahead electricity market modeling and simulation. This algorithm can not only get rid of the abovementioned unrealistic assumptions, but also cope with the Markov decision-making process with continuous state and action sets just like the real electricity market. Meanwhile, the time complexity of our proposed model is only O(n). The simulation result of employing the proposed model in the double-side day-ahead electricity market shows the superiority of our approach in terms of participant's profit or social welfare compared with traditional reinforcement learning methods.

Bacteria Foraging Reinforcement Learning for Risk-Based Economic Dispatch via Knowledge Transfer

“…Conventional optimization algorithms, such as nonlinear programming [17], gradient decent method [18], interior point method [19], and the Newton method [20], may be easily trapped in a local optimum. Besides, an accurate system model and appropriate feasible initial solutions are needed to achieve a good application effect, based on which software (Gurobi [21] and CPLEX [22]) is not flexible enough and inapplicable for some complex problems.…”

Online Area Load Modeling in Power Systems Using Enhanced Reinforcement Learning

“…In applications involving the online control or dispatch of power systems, reinforcement learning (RL) methods have exhibited remarkable capabilities for performing online calculations [21][22][23]. RL is based on the concept of trial and error and explicitly considers the problem of a goal-directed agent interacting with an uncertain environment [24].…”

Day-Ahead Market Modeling for Strategic Wind Power Producers under Robust Market Clearing

“…This work intends to propose a LSCAC-based day-head EM modeling approach to simulate this dynamic multi-WPP decision-making process. Hence, similar to Reference [41], some necessary definitions are organized as follows:…”

“…Owning to this fact of incomplete and imperfect information in the day-ahead EM [41], strategic WPPs must dynamically improve their profits through repeatedly bidding in day-ahead EM, which is actually a dynamic multi-participant decision-making process. This work intends to propose a LSCAC-based day-head EM modeling approach to simulate this dynamic multi-WPP decision-making process.…”

“…According to [41], the day-ahead EM is actually a dynamic complex system in which dynamic (direct and indirect) interactions exist among all participants. When considering WPPs' strategic behaviors in the day-ahead EM, on one hand, the MCM of the ISO should be modified in order to accommodate the deviations caused by real time wind power output uncertainties [10,19,20], while on the other hand, EM modeling approaches should be proposed to help with obtaining WPPs' reasonable strategies under different BMs.…”