Reinforcement Learning Based EV Charging Management Systems–A Review

Abstract

To mitigate global warming and energy shortage, integration of renewable energy generation sources, energy storage systems, and plug-in electric vehicles (PEVs) have been introduced in recent years. The application of electric vehicles (EV) in the smart grid has shown a significant option to reduce carbon emission. However, due to the limited battery capacity, managing the charging and discharging process of EV as a distributed power supply is a challenging task. Moreover, the unpredictable nature of renewable energy generation, uncertainties of plug-in electric vehicles associated parameters, energy prices, and the time-varying load create new challenges for the researchers and industries to maintain a stable operation of the power system. The EV battery charging management system plays a main role in coordinating the charging and discharging mechanism to efficiently realize a secure, efficient, and reliable power system. More recently, there has been an increasing interest in data-driven approaches in EV charging modeling. Consequently, researchers are looking to deploy model-free approaches for solving the EV charging management with uncertainties. Among many existing model-free approaches, Reinforcement Learning (RL) has been widely used for EV charging management. Unlike other machine learning approaches, the RL technique is based on maximizing the cumulative reward. This paper reviews the existing literature related to the RL-based framework, objectives, and architecture for the charging coordination strategies of electric vehicles in the power systems. In addition, the review paper presents a detailed comparative analysis of the techniques used for achieving different charging coordination objectives while satisfying multiple constraints. This paper also focuses on the application of RL in EV coordination for research and development of the cutting-edge optimized energy management system (EMS), which are applicable for EV charging.

Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges

“…In [121]- [123], various RL techniques are studied to design EV charging policies to deal with the randomness in the arrival and departure time of an EV. See [124] for a review on RL-based EV charging management systems. References [125]- [127] adopt DQN and DDPG to learn the charging/discharging strategy for controlling battery systems considering unknown degradation models.…”

Optimal load management strategy for large electric vehicle charging stations with undersized charger clusters

“…The readers may refer to the review papers such as Ref. [19][20][21] for further theoretical analysis on alternative control schemes. In this section, we will mention few representative examples to give a general overview on state-of-the-art strategies and demonstrate the need for an original formulation to deal with the novel problem that is defined in Section 1.…”

“…For a detailed review on machine learning approaches, the interested readers are advised to refer to Ref. [21,27]. The ability to find a near optimal solution without the execution of an optimisation problem is the main advantage of learning-based strategies; therefore, they are applied mostly to deal with uncertainty in complex systems.…”

Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning

“…The problem can be formulated as a 0-AIR MDP where the private variable is a binary variable indicating whether we have chosen to stop or not. Other examples include those where the agent only influences energy efficiency, such as in the hybrid vehicle problem (Shahamiri 2008) and the electric vehicles charging problem (Abdullah, Gastli, and Ben-Brahim 2021). In the former problem, the agent controls the vehicle to use either the gas engine or the electrical motor at each time step, with the goal to minimize gas consumption; its actions do not impact the driver's behavior.…”