Bi-level energy management strategy for power-split plug-in hybrid electric vehicles: A reinforcement learning approach for prediction and control

Abstract

The implementation of an energy management strategy plays a key role in improving the fuel economy of plug-in hybrid electric vehicles (PHEVs). In this article, a bi-level energy management strategy with a novel speed prediction method leveraged by reinforcement learning is proposed to construct the optimization scheme for the inner energy allocation of PHEVs. First, the powertrain transmission model of the PHEV in a power-split type is analyzed in detail to obtain the energy routing and its crucial characteristics. Second, a Q-learning (QL) algorithm is applied to establish the speed predictor. Third, the double QL algorithm is introduced to train an effective controller offline that realizes the optimal power distribution. Finally, given a reference battery's state of charge (SOC), a model predictive control framework solved by the reinforcement learning agent with a novel speed predictor is proposed to build the bi-level energy management strategy. The simulation results show that the proposed method performs with a satisfying fuel economy in different driving scenarios while tracking the corresponding SOC references. Moreover, the calculation performance also implies the potential online capability of the proposed method.

Energy Management Systems’ Modeling and Optimization in Hybrid Electric Vehicles

“…The output layer produces the final predictions based on the processed input data, which contains the neurons that transmit the outputs from the hidden layers [3,101,104]. Hidden layers are responsible for processing the input data and extracting relevant features, and the number of hidden layers and the number of neurons in each layer can vary depending on the complexity of the problem and the desired level of accuracy [96,105,106]. Neural networks learn the properties of the data and make predictions through the connections between these layers and the activation functions of the neurons [96,103].…”

Machine Learning and Optimization in Energy Management Systems for Plug-In Hybrid Electric Vehicles: A Comprehensive Review

“…SOC optimization or fuel reduction DDPG [5], DRL [40], DP [41], RL [42,43], DP, NN-based EMS [44], LTV-SMPC and PMP-stochastic MPC [45] Projected interior point method [3], LQP [45] DRL, rule-based, DDPG [40], Gaussian mixture model, SDP [41], QL, MPC [42], WF2SLOA [46], C/GMRES, BO [18], LQP, MPC, PMP [45] Hierarchical EMS [5], Hybrid EMS with torque split between the ICE and ESS [46], MPC EMS with non-linear losses [3] 16.34% of fuel savings [5], fuel economy improvement by 0.55% [40], LTV-SMPC and PMP-SMPC increase fuel economy by 8.79% and 14.42% respectively Prediction LSTM [5], Markov chain and LSTM [45] Power split with NN-based EMS [44] Speed [5,40,42] Prediction of mode and power split 2% higher compared to DP [44] Real-time power distribution MPC [5,42], C/GMRES [47] Polynomial fitting approx.…”

“…(methodical derivatives) [47] MPC, RL [42] Bi-level EMS [42], data-driven calibration [47] Engine operating time is reduced by up to 2.96% [41] DDPG The EMS formulation requires understanding the problem, i.e., identifying the power train configuration and the power flow among components. Generally, the EMS relies on several concepts regarding power and energy control strategies [31].…”

A Predictive Energy Management Strategy for Heavy Hybrid Electric Vehicles Based on Adaptive Network-Based Fuzzy Inference System-Optimized Time Horizon

“…Therefore, combining vehicle velocity prediction with MPC can be a way to optimize the energy management strategy for HEVs [37][38][39][40][41]. In ref.…”