MAPDP: Cooperative Multi-Agent Reinforcement Learning to Solve Pickup and Delivery Problems

Abstract

Cooperative Pickup and Delivery Problem (PDP), as a variant of the typical Vehicle Routing Problems (VRP), is an important formulation in many real-world applications, such as on-demand delivery, industrial warehousing, etc. It is of great importance to efficiently provide high-quality solutions of cooperative PDP. However, it is not trivial to provide effective solutions directly due to two major challenges: 1) the structural dependency between pickup and delivery pairs require explicit modeling and representation. 2) the cooperation between different vehicles is highly related to the solution exploration and difficult to model. In this paper, we propose a novel multi-agent reinforcement learning based framework to solve the cooperative PDP (MAPDP). First, we design a paired context embedding to well measure the dependency of different nodes considering their structural limits. Second, we utilize cooperative multi-agent decoders to leverage the decision dependence among different vehicle agents based on a special communication embedding. Third, we design a novel cooperative A2C algorithm to train the integrated model. We conduct extensive experiments on a randomly generated dataset and a real-world dataset. Experiments result shown that the proposed MAPDP outperform all other baselines by at least 1.64\% in all settings, and shows significant computation speed during solution inference.

Distilling Autoregressive Models to Obtain High-Performance Non-autoregressive Solvers for Vehicle Routing Problems with Faster Inference Speed

“…With the emergence of deep learning, numerous NN-based model have been proposed in recent years to solve VRPs (Fu, Qiu, and Zha 2021;Zong et al 2022;Zhang et al 2022). These models can be generally categorized into neural improvement type and neural construction type.…”

Siamese-Discriminant Deep Reinforcement Learning for Solving Jigsaw Puzzles with Large Eroded Gaps

“…Reinforcement Learning (RL) has witnessed a great number of successes in combinatorial optimization (CO) (Bengio, Lodi, and Prouvost 2021;Woo, Lee, and Cho 2022;Chen and Tian 2019;Zong et al 2022;Bai et al 2021), robotics (Tomar, Sathuluri, and Ravindran 2019), natural language processing (Li, Kiseleva, and De Rijke 2019) and computer vision (Kim et al 2021). Combination of RL and graph neural network also flourished in recent years (Vesselinova et al 2020).…”

A review on learning to solve combinatorial optimisation problems in manufacturing

“…In ref. [169], the multi‐agent RL was explored to solve PDTSP with capacity constraint. In a recent work by Ma et al.…”

“…In ref. [169], the multi-agent RL was explored to solve PDTSP with capacity constraint. In a recent work by Ma et al [72], the neural neighbourhood search (N2S) framework was proposed which was the first learning-based solver that outstrips the well-known LKH3 solver [176] on solving PDTSP and its variant with Last-In-Fist-Out (LIFO) constraint on the synthesised dataset.…”