Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships

Abstract

Hybrid fuel cell and battery propulsion systems have the potential to offer improved emission performance for coastal ships with access to H 2 replenishment and battery charging infrastructures in ports. However, such systems could be constrained by high power source degradation and energy costs. Cost-effective energy management strategies are essential for such hybrid systems to mitigate the high costs. This article presents a Double Q reinforcement learning based energy management system for such systems to achieve near-optimal average voyage cost. The Double Q agent is trained using stochastic power profiles collected from continuous monitoring of a passenger ferry, using a plug-in hybrid fuel cell and battery propulsion system model. The energy management strategies generated by the agent were validated using another test dataset collected over a different period. The proposed methodology provides a novel approach to optimal use hybrid fuel cell and battery propulsion systems for ships. The results show that without prior knowledge of future power demands, the strategies can achieve near-optimal cost performance (96.9%) compared to those derived from using dynamic programming with the equivalent state space resolution.

An Intelligent Energy Management Framework for Hybrid-Electric Propulsion Systems Using Deep Reinforcement Learning

“…This article aims to propose a generic energy management framework for hybrid-electric propulsion systems. The energy management framework is based on Deep Reinforcement Learning (DRL) to deal with the high-dimensional control of multiple power sources operating under uncertainties, extending the authors' previous work in [2,6]. The proposed energy management framework is demonstrated through a case study of a typical coastal ferry with continuous monitoring of its operational status [8].…”

“…Consequently, they can only deliver the desired performance within the specific load profiles used to calibrate the EMS. Nevertheless, offline optimisation-based EMS can be used as a benchmark to evaluate the effectiveness of other online EMS, as long as there was a priori knowledge of the complete load profile beforehand [6]. In contrast, an online strategy, such as Equivalent Consumption Minimisation Strategies (ECMS) and Model Predictive Control (MPC), does not require a priori knowledge of the load profile and is typically formulated as an instantaneous optimisation problem for implementation with limited computational resources in real-time operations [10].…”

“…Although such an approach is valid, the fitted control function originally derived from dynamic programming is computationally expensive and is constrained by the 'curse of dimensionality' [20], making it impracticable for systems with high-dimensional and/or continuous state and actions spaces. In contrast, a Reinforcement Learning (RL) agent learns an optimal or near-optimal energy management strategy by continuously interacting with its 'Environment' (including both the hybrid system model and historical load profiles) [6]. Effort from [21] solved the optimal energy management problem of a hybrid-electric vehicle using RL in discrete state and action spaces, with their RL agents trained with a limited number of load profiles.…”

“…Effort from [21] solved the optimal energy management problem of a hybrid-electric vehicle using RL in discrete state and action spaces, with their RL agents trained with a limited number of load profiles. Wu et al [6] trained their RL agent using large-scale continuous monitoring data, addressing the problem of function over-estimations when the environment is highly scholastic. Wu et al [22] extended the state space to be continuous while the action space is still discrete.…”

Influence of the Hull Shape on the Energy Demand of a Small Inland Vessel with Hybrid Propulsion

“…At the same time, it is necessary for the propulsion system to work as efficiently as possible. Achieving the optimum operating parameters of the propulsion is possible through the use of various energy management strategies, accroding to [7] - [9].…”