Online Data-Driven Energy Management of a Hybrid Electric Vehicle Using Model-Based Q-Learning

Abstract

The energy management strategy of a hybrid electric vehicle directly determines the fuel economy of the vehicle. As a supervisory control strategy to divide the required power into its multiple power sources, engines and batteries, many studies have been conducting using rule-based and optimization-based approaches for energy management strategy so far. Recently, studies using various machine learning techniques have been conducted. In this paper, a novel control framework implementing Model-based Q-learning is developed for the optimal control problem of hybrid electric vehicles. As an online energy management strategy, a new approach could learn the characteristics of a current given driving environment and adaptively change the control policy through learning. Especially, for the proposed algorithm, the internal powertrain environment and external driving environment are separated so they can be learned via the reinforcement learning framework, which results in a simpler and more intuitive control strategy that can be explained using the vehicle state approximation model. The proposed algorithm is tested and verified through simulations, and the simulation results present near optimal solution. The simulation results are compared with conventional rule-based strategies and optimal control solutions acquired from Dynamic Programming.INDEX TERMS Hybrid electric vehicle, optimal control, power management, Q-learning, reinforcement learning.

Model-Based Reinforcement Learning for Eco-Driving Control of Electric Vehicles

“…In this study, to solve the optimal control problem, a novel eco-driving strategy utilizing MBRL was developed on the basis of a previous study on MBRL for the HEV control case study in [21]. In (17), the state variable…”

“…whereĝ k , andv k+1 can be determined based on the vehicle powertrain model. Alternatively,ĝ k , andv k+1 can be determined based on approximation model for keeping modelfree characteristic of reinforcement learning(see [21]), that approximation could be done based on the experience as shown in following equation:…”

“…MBRL is a methodology that approximates the environment, including the system dynamics; thus, learning can be conducted with guaranteed stability [18], or few interactions [19]. In the case of vehicle control, MBRL was successfully applied to the optimal control problem of energy management of HEVs in our previous studies [20], [21]. The contribution of the present study is as follows: We developed a new algorithm for the eco-driving control problem using the reinforcement learning approach, and through this, we confirmed that the reinforcement learning method can be well applied to the eco-driving problem.…”

Battery Energy Management of Autonomous Electric Vehicles Using Computationally Inexpensive Model Predictive Control

“…However, the interactions between the road participants in the traffic cannot be modelled in deterministic way. To handle the such uncertainties in the traffic prediction, stochastic DP [12], Q-learning-based PMP [13], and reinforcement learning [14,15] have been considered, but the analysis on the practical aspects like computational tractability is omitted. Additionally, since these methods are based on Markov decision process, where the relationship between the control action and probability are pre-determined, these methods cannot ensure robustness when the vehicle is in an unexpected situation [16].…”

“…Due to the nonconvexity in P b , the control fluctuates a lot to search the optimum point at every step, and is sometimes stuck at the local minimum points, as shown in the middle plot of Figure 6. In contrast, by exploiting the quadratic form (15), the computation time at each step is significantly reduced compared to the original form, as shown in the bottom plot of Figure 6. Therefore, it can be concluded that the nonlinear cost (10a) can be replaced with the quadratic cost (15) without sacrificing too much performance in terms of the battery SOC reduction.…”