Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learning

Abstract

A fundamental question in any peer-to-peer ridesharing system is how to, both effectively and efficiently, dispatch user's ride requests to the right driver in real time. Traditional rule-based solutions usually work on a simplified problem setting, which requires a sophisticated hand-crafted weight design for either centralized authority control or decentralized multi-agent scheduling systems. Although recent approaches have used reinforcement learning to provide centralized combinatorial optimization algorithms with informative weight values, their single-agent setting can hardly model the complex interactions between drivers and orders. In this paper, we address the order dispatching problem using multi-agent reinforcement learning (MARL), which follows the distributed nature of the peer-to-peer ridesharing problem and possesses the ability to capture the stochastic demand-supply dynamics in largescale ridesharing scenarios. Being more reliable than centralized approaches, our proposed MARL solutions could also support fully distributed execution through recent advances in the Internet of Vehicles (IoV) and the Vehicle-to-Network (V2N). Furthermore, we adopt the mean field approximation to simplify the local interactions by taking an average action among neighborhoods. The mean field approximation is capable of globally capturing dynamic demand-supply variations by propagating many local interactions between agents and the environment. Our extensive experiments have shown the significant improvements of MARL order dispatching algorithms over several strong baselines on the gross merchandise volume (GMV), and order response rate measures. Besides, the simulated experiments with real data have also justified that our solution can alleviate the supply-demand gap during the rush hours, thus possessing the capability of reducing traffic congestion.

CoRide

“…For all learning methods, following [13], we run 20 episodes for training, store the trained model periodically, and conduct the evaluation on the stored model with 5 random seeds. We compare the performance of different models regarding two criteria, including ADI, computed as the total income in a day, and ORR, calculated by the number of orders taken divided by the number of orders generated.…”

“…It only assigns idle vehicles with available orders randomly at each timestep. • DQN: Li et al [13] conducted action-value function approximation based on Q-network. The Q-network is parameterized by a MLP with four hidden layers and we adopt the ReLU activation between hidden layers and to transform the final linear output of Q-network.…”

Multi-Objective Predictive Taxi Dispatch via Network Flow Optimization

“…RL is intended to capture the interactions between a large volume of vehicles in an adaptive manner. However, due to the curse of dimensionality, in practice, it is used in conjunction with an approximation technique, which often degrades the performance of this approach in large-scale fleet management [24,25]. RL methods also require a substantial amount of data to learn an efficient dispatch policy by capturing how to utilize various factors in a given transportation system [18,[26][27][28].…”

“…For example, the allocation of the ride request can be modeled by a combinatorial optimization problem and the acceptance of the allocated request can be modeled by the probability distribution [13]. Moreover, an independent and cooperative ride request allocating algorithm was also proposed [24]. In [38], an RL-based algorithm that can allocate the large-scale ride requests in real-time was proposed.…”

Multi-Agent Reinforcement Learning for Order-dispatching via Order-Vehicle Distribution Matching

“…However, for centralized approaches, a critical issue is the potential "single point of failure" [18], i.e., the failure of the centralized authority control will fail the whole system [16]. Another two related work using multi-agent to learn order-dispatching is based on mean-eld MARL [13] and knowledge transferring [35].…”