Scalable Multiagent Driving Policies For Reducing Traffic Congestion

Abstract

Traffic congestion is a major challenge in modern urban settings. The industry-wide development of autonomous and automated vehicles (AVs) motivates the question of how can AVs contribute to congestion reduction. Past research has shown that in small scale mixed traffic scenarios with both AVs and human-driven vehicles, a small fraction of AVs executing a controlled multiagent driving policy can mitigate congestion. In this paper, we scale up existing approaches and develop new multiagent driving policies for AVs in scenarios with greater complexity. We start by showing that a congestion metric used by past research is manipulable in open road network scenarios where vehicles dynamically join and leave the road. We then propose using a different metric that is robust to manipulation and reflects open network traffic efficiency. Next, we propose a modular transfer reinforcement learning approach, and use it to scale up a multiagent driving policy to outperform human-like traffic and existing approaches in a simulated realistic scenario, which is an order of magnitude larger than past scenarios (hundreds instead of tens of vehicles). Additionally, our modular transfer learning approach saves up to 80% of the training time in our experiments, by focusing its data collection on key locations in the network. Finally, we show for the first time a distributed multiagent policy that improves congestion over human-driven traffic. The distributed approach is more realistic and practical, as it relies solely on existing sensing and actuation capabilities, and does not require adding new communication infrastructure.

DM$^2$: Distributed Multi-Agent Reinforcement Learning for Distribution Matching

“…To do so, the agents are provided a mixed reward: part cost function for minimizing individual distribution mismatch, part environment reward. This approach has been shown to be effective in balancing individual preferences with shared objectives in multi-agent RL [6,5]. The individual agent policies are learned by independently updating each agent's policy using an on-policy RL algorithm of choice.…”

Learning a Robust Multiagent Driving Policy for Traffic Congestion Reduction

“…Most of these past successful driving policies controlled AVs in a centralized manner, where a single controller simultaneously processes all available sensing information and sends driving commands to the AVs. More recent efforts focused on developing decentralized driving policies which might be harder to learn, but are considered a more realistic option for real-world deployment, as they mostly rely on local sensing and actuation capabilities [6,27]. This paper continues the line of research on decentralized policies but aims to develop one that is robust to real-world traffic conditions of practical interest.…”

“…Congestion reduction driving policies can either be centralized, controlling all vehicles simultaneously based on global system information, or decentralized, where each vehicle is controlled independently based on its local observations. Decentralized policies with no vehicle-to-vehicle communication are most realistic, since they mostly rely on local sensing and actuation capabilities [6,26], and are therefore the focus of this paper. To facilitate data and computational efficiency and reduce the risk of overfitting, all AVs learn and execute a single, shared driving policy, so that the resulting number of learned parameters is relatively small.…”

“…Leveraging the Flow framework that interacts with SUMO, this paper focuses on developing a decentralized driving policy using the Proximal Policy Optimization (PPO) algorithm [17]. In this paper we use an observation and reward design for each AV modeled after that used by Cui et al [6]. The observation for each AV is listed as follows:…”

“…A common form of traffic congestion on highways is stop-and-go waves, which have been shown in field experiments to emerge when vehicle density exceeds a critical value [19]. Past research has shown that in human-driven traffic, a small fraction of automated or autonomous vehicles (AVs) executing a controlled multiagent driving policy can mitigate stop-and-go waves in simulated and real-world scenarios, roughly double the traffic speed and increase throughput by about 16% [6,12,18,27,29]. Frequently, the highest-performing policies are those learned by deep reinforcement learning (DRL) algorithms, rather than hand-coded or model-based control policies.…”