Interterminal Truck Routing Optimization Using Cooperative Multiagent Deep Reinforcement Learning

Abstract

Many ports worldwide continue to expand their capacity by developing a multiterminal system to catch up with the global containerized trade demand. However, this expansion strategy increases the demand for container exchange between terminals and their logistics facilities within a port, known as interterminal transport (ITT). ITT forms a complex transportation network in a large port, which must be managed efficiently given the economic and environmental implications. The use of trucks in ITT operations leads to the interterminal truck routing problem (ITTRP), which has been attracting increasing attention from researchers. One of the objectives of truck routing optimization in ITT is the minimization of empty-truck trips. Selection of the transport order (TO) based on the current truck location is critical in minimizing empty-truck trips. However, ITT entails not only transporting containers between terminals operated 24 h: in cases where containers need to be transported to a logistics facility within operating hours, empty-truck trip cost (ETTC) minimization must also consider the operational times of the transport origin and destination. Otherwise, truck waiting time might be incurred because the truck may arrive before the opening time of the facility. Truck waiting time seems trivial, but it is not, since thousands of containers move between locations within a port every day. So, truck waiting time can be a source of ITT-related costs if it is not managed wisely. Minimization of empty-truck trips and truck waiting time is considered a multiobjective optimization problem. This paper proposes a method of cooperative multiagent deep reinforcement learning (RL) to produce TO truck routes that minimize ETTC and truck waiting time. Two standard algorithms, simulated annealing (SA) and tabu search (TS) were chosen to assess the performance of the proposed method. The experimental results show that the proposed method represents a considerable improvement over the other algorithms.

Trailer allocation and truck routing using bipartite graph assignment and deep reinforcement learning

“…They considered the origins and destinations list provided beforehand and the method they used to solve the homogeneous vehicle PDP is DQN with experience replay memory. They expanded their method using cooperative multi‐agent DRL in their following work (Adi et al, 2021).…”

The Effect of the Number of Hidden Layers on The Performance of Deep Q-Network for Traveling Salesman Problem

“…Reference [7], in their study, utilized the DQN algorithm to address shipping and route issues for autonomous robots. Research conducted by [8] used DQN to solve the truck routing problem between terminals to minimize the total cost incurred. However, in recent years, machine learning advancements have been utilized to solve TSP-related problems.…”

Solving the Inter-Terminal Truck Routing Problem for Delay Minimization Using Simulated Annealing with Normalized Exploration Rate

“…Decisions involving ITT encompass multimodal transportation such as barges, rails, and trucks. However, because of the flexibility and common use of trucks for inland transportation, it is important to model truck-specific routing problems in inter-terminal operations [3,6,20]. Heilig et al [4] introduced the ITTRP to model a container pick-up delivery problem of trucks by considering truck service time and constant delay percontainer penalty into the optimization objective while putting an emphasis on reducing empty truck trips (ETT).…”

“…Further study by Adi et al [3] proposed a learning-based approach by applying deep reinforcement learning to ITTRP. Moreover, this study extends the model to cooperative multi-agent deep reinforcement learning [20] where each truck represents a single agent. Closely related to ITTRP is the recent study by Baals et al [14] in minimizing earliness-tardiness costs in supplier networks for a just-in-time truck routing problem (TRP-JIT).…”