Context aware Q-Learning-based model for decision support in the negotiation of energy contracts

Abstract

Automated negotiation plays a crucial role in the decision support for bilateral energy transactions. In fact, an adequate analysis of past actions of opposing negotiators can improve the decision-making process of market players, allowing them to choose the most appropriate parties to negotiate with in order to increase their outcomes. This paper proposes a new model to estimate the expected prices that can be achieved in bilateral contracts under a specific context, enabling adequate risk management in the negotiation process. The proposed approach is based on an adaptation of the Q-Learning reinforcement learning algorithm to choose the best scenario (set of forecast contract prices) from a set of possible scenarios that are determined using several forecasting and estimation methods. The learning process assesses the probability of occurrence of each scenario, by comparing each expected scenario with the real scenario. The final chosen scenario is the one that presents the higher expected utility value. Besides, the learning method can determine which is the best scenario for each context, since the behaviour of players can change according to the negotiation environment. Consequently, these conditions influence the final contract price of negotiations. This approach allows the supported player to be prepared for the negotiation scenario that is the most probable to represent a reliable approximation of the actual negotiation environment.

Deep reinforcement learning for acceptance strategy in bilateral negotiations

“…Rodriguez et al [32] study bilateral negotiations for electricity market energy contracts. They introduce a context aware Q-learning approach for energy contracts.…”

The importance of context-dependent learning in negotiation agents

“…A colloquial definition of the negotiation context can be found in [20]. According to Rodriguez-Fernandez et al, the negotiation context refers to characteristics or circumstances under which the negotiation process occurs.…”

“…In [9], agents make offers and take decisions (accept/reject) based in a simple negotiation domain, summarized in reservation value and discount factor, but agents do not consider the dynamics of the context (or domain) to compute this variables. In [20], context obtains a clearer attention from the authors. Still, learning capabilities are used to model and select a forecast method to some contextual variables, but there is no mention to the strategy or policy used by an agent is going to act or concede during a particular negotiation given the context he is in.…”

A Q-Cube Framework of Reinforcement Learning Algorithm for Continuous Double Auction among Microgrids

“…Salehizadeh et al [32] proposed a fuzzy QL approach in the presence of renewable resources under both normal and stressful cases. The authors in [33] introduced the concept of scenario extraction into a QL-based energy trading model for decision support.…”