An Online Reinforcement Learning Approach for Dynamic Pricing of Electric Vehicle Charging Stations

Abstract

The global market share of electric vehicles (EVs) is on the rise, resulting in a rapid increase in their charging demand in both spatial and temporal domains. A remedy to shift the extra charging loads at peak hours to off-peak hours, caused by charging EVs at public charging stations, is an online pricing strategy. This paper presents a novel combinatorial online pricing strategy that has been established upon a reward-based model to prevent network instability and power outages. In the proposed solution, the utility provides incentives to the charging stations for their contributions in the EVs charging load shifting. Then, a constraint optimization problem is developed to minimize the total charging demand of the EVs during peak hours. To control the EVs charging demands in supporting utility's stability and increasing the total revenue of the charging stations, treated as a multi-agent framework, an online reinforcement learning model is developed which is based on the combination of an adaptive heuristic critic and recursive least square algorithm. The effective performance of the proposed model is validated through extensive simulation studies such as qualitative, numerical, and robustness performance assessment tests. The simulation results indicate significant improvement in the robustness and effectiveness of the proposed solution in terms of utility's power saving and charging stations' profit. INDEX TERMS Electric vehicles, charging stations, pricing strategy, reinforcement learning. Nomenclature Abbreviations ACO Ant colony optimization. AHC Adaptive heuristic critic. BSS Battery storage system. CDP Coordinated dynamic pricing. CS Charging station. EV Electric vehicle. M DP Markov decision process. RL Reinforcement learning. RLS Recursive least square. Parameters α EVs load index. ∆t Time slot.

Reinforcement Learning Based EV Charging Management Systems–A Review

“…Some other studies focus on using renewable energy sources and their inflexibility in producing energy, monitored their energy production to achieve better managing of the distributed energy systems. Other studies maintained the battery SOC of the EVs above a certain threshold by monitoring the battery SOC to be able to produce the appropriate actions [64], [65], [66], [67], [68], [69], [70], [71]. Other parameters were used less frequently in studies as state parameters, as they were not related to the frequently desired objectives [72] and [73].…”

“…The authors in papers [61]- [72], [74], [75], [76], [77], [65], [78], [79], [80]- [81], [82], [83], [84]- [67], [85] - [86], and [87]- [69], used the charging or discharging cost as a state parameter. However, other operational costs, like waiting cost and traveling, were used mainly in two of the surveyed papers [88] and [67].…”

“…Another parameter that was used as a state parameter by many of the proposed methods in [96], [60], [97], [99], [74], [90], [76], [77], [65], [78], [79], [101], [102]- [103], [105], [82], [83]- [84], [95], [85], [107], [87], [69], and [70]- [71], is the EVs charging demand satisfaction. These proposed methods aimed at controlling the energy cost, reducing the prices for the customers, or even increasing the charger's profit.…”

“…The relation between the demand and the prices is existent in any business, so, eventually, it exists in the energy distribution systems. Some other proposed methods ( [97], [99], [77], [103], [105], [85], [107], [69], and [70]) aimed to better control the load profile and used the charging demand as a state parameter, as the load of the system increases when EVs demand electricity.…”

“…The studies [77], [69], and [70] attempted to manage the load on the transformers at the EVCS in order to maintain and ensure that no failures happen in the charging system. The papers [108], [109], and [63] proposed single-agent RL solutions targeting balancing the power between the generation side and the consumption side.…”