Distributed Optimization of Solar Micro-grid Using Multi Agent Reinforcement Learning

Abstract

In the distributed optimization of micro-grid, we consider grid connected solar micro-grid system which contains a local consumer, a solar photovoltaic system and a battery. The consumer as an agent continuously interacts with the environment and learns to take optimal actions. Each agent uses a model-free reinforcement learning algorithm, namely Q Learning, to optimize the battery scheduling in dynamic environment of load and available solar power. Multiple agents sense the states of the environment components and make collective decisions about how to respond to randomness in load, intermittent solar power using a Multi-Agent Reinforcement Learning algorithm, called Coordinated Q Learning (CQL). The goals of each agent are to increase the utility of the battery and solar power in order to achieve the long term objective of reducing the power consumption from grid.

Application of a Gradient Descent Continuous Actor-Critic Algorithm for Double-Side Day-Ahead Electricity Market Modeling

“…The comparison result shows that with Boltzmann ε-greedy decision method, the participants in the day-ahead market can receive more profits after sufficient learning iterations, because the value of the temperature variable in the Boltzmann action choosing probability distribution function of every agent (participant) can be adjusted as the iteration proceeds. Similar studies in electricity market simulation and other areas can also be found in [26][27][28][29][30][31][32][33][34][35][36].…”

“…Taking provisions (1)-(4) into consideration, the methods both in [8][9][10][11][12][13][14][15][16][17][18][20][21][22][23] and in [19,[24][25][26][27][28][29][30][31][32][33] are not quite suitable for modeling and simulating the practical day-ahead electricity market exactly, the reason of which is that in the real day-ahead electricity market, every participant can adjust its bidding strategy within a continuous interval of values, but the literatures suitable for provisions (1)- (3) have assumed that the sets for alternative actions (e.g., bidding strategies) or potential states (e.g., historical MCP) are discrete.…”

“…Therefore, each of the GenCOs and DisCOs can be considered as an agent [19,[24][25][26][27][28][29][30][31][32][33]. In this paper, without loss of generality, it is assumed that each GenCO has only one generation unit and submits one bid curve for each time interval of the next day, and considers the profit obtained through bidding in corresponding time interval of the next day, the same as each DisCO.…”

“…As listed in Section 1.2, every participant (a GenCO or a DisCO) has no idea about its rivals' strategies historically and currently, what it knows is the information about historical MCPs. Literatures [19,25,26,29,33] have proposed that an agent-based GenCO (or DisCO) learns from the MCP (or LMP) of the last round market competition calculated and published by the ISO to decide which bidding strategy can be used in current market bidding competition in order to pursue its own profit maximization.…”

“…Based on the viewpoints expressed in [19,25,26,29,33], this paper proposes that an agent-based GenCO or DisCO participating in a double-side day-ahead electricity market learns from the historical MCP, calculated and published by the ISO yesterday (for today), to decide which bidding strategy can be applied for the next day in order to pursue its own profit maximization. Hence, there are some definitions described in this paper as follows:…”