Deep Policy Dynamic Programming for Vehicle Routing Problems

Abstract

Routing problems are a class of combinatorial problems with many practical applications. Recently, end-to-end deep learning methods have been proposed to learn approximate solution heuristics for such problems. In contrast, classical dynamic programming (DP) algorithms guarantee optimal solutions, but scale badly with the problem size. We propose Deep Policy Dynamic Programming (DPDP), which aims to combine the strengths of learned neural heuristics with those of DP algorithms. DPDP prioritizes and restricts the DP state space using a policy derived from a deep neural network, which is trained to predict edges from example solutions. We evaluate our framework on the travelling salesman problem (TSP), the vehicle routing problem (VRP) and TSP with time windows (TSPTW) and show that the neural policy improves the performance of (restricted) DP algorithms, making them competitive to strong alternatives such as LKH, while also outperforming most other 'neural approaches' for solving TSPs, VRPs and TSPTWs with 100 nodes.

Exploring the Frontier of Combinatorial Optimization and Machine Learning: Applications to Vehicle Routing and Support Vector Machines

“…Underpinned by enhancements in hardware and artificial intelligence research over the last years, the development of deep NNs made them relevant to a wide range of difficult combinatorial optimization problems, such as SAT, Minimum Vertex Cover, and Maximum Cut [142,143]. When applied to solve CVRPs, these networks are usually combined with reinforcement learning (RL [144, 145]) or typically used for node classification or edge prediction [146,147]. Despite extensive research, GNNs for directly solving CVRPs remain limited to small problem instances with up to 100 customers and generally do not compare favorably with classic optimization methods (exact or heuristic) in terms of solution quality.…”

“…In this study, in particular, we aim to learn and use relatedness information in the LS and crossover operators of HGS to improve this stateof-the-art method substantially. We capitalize upon the work of [146], which trained a GNN to predict occurrence probabilities of edges in high-quality solutions (i.e., heatmap), and used this information to sparsify the underlying graph and accelerate related solution procedures. Instead, we leverage the heatmaps as a source of relatedness information to define neighborhood restrictions in the LS and possible re-connection points in the crossover.…”

“…This definition is general: in the simplest setting, relatedness could be the inverse of distance, i.e., ϕ(i, j) = 1/d ij . In a more informed setting, we can instead consider defining ϕ(i, j) as the output of a graph neural network (GNN) as seen in [146], predicting the probability of occurrence of an edge in a high-quality solution. Probabilities of this kind are typically called heatmaps.…”

“…In light of this, an emerging line of research embraces neural networks (NN) as (sub)routines to solve CO problems. For example, [146] presented a graph NN that predicts the probability of each edge participating in a highquality solution. All edges whose probabilities are below a threshold are then used to prune the search space of solutions in applications for the capacitated vehicle routing problem (CVRP) and the traveling salesperson problem (TSP).…”

“…All experiments are run on a single thread of an Intel Gold 6148 Skylake 2.4 GHz processor with 40 GB of RAM and NVIDIA Tesla P100 Pascal (12 G memory), running CentOS 7.8.2003. Unless otherwise stated, we use the original parameters defined for HGS in [136] and the GNN in [146]. To achieve fast convergence, we set smaller values for the population-size parameters in HGS: µ = 12 and λ = 20.…”