A Multi-AUV Maritime Target Search Method for Moving and Invisible Objects Based on Multi-Agent Deep Reinforcement Learning

Abstract

Target search for moving and invisible objects has always been considered a challenge, as the floating objects drift with the flows. This study focuses on target search by multiple autonomous underwater vehicles (AUV) and investigates a multi-agent target search method (MATSMI) for moving and invisible objects. In the MATSMI algorithm, based on the multi-agent deep deterministic policy gradient (MADDPG) method, we add spatial and temporal information to the reinforcement learning state and set up specialized rewards in conjunction with a maritime target search scenario. Additionally, we construct a simulation environment to simulate a multi-AUV search for the floating object. The simulation results show that the MATSMI method has about 20% higher search success rate and about 70 steps shorter search time than the traditional search method. In addition, the MATSMI method converges faster than the MADDPG method. This paper provides a novel and effective method for solving the maritime target search problem.

Consistent Epistemic Planning for Multiagent Deep Reinforcement Learning

“…Target search and tracking (S&T) is a sustained and focused research topic in recent years [1], [2], [5], [6], [8]- [11], which refers to detecting an object of interest in a search space [1] and tracking it. The related research can be divided into two components -1) to design a filter using a series of past observations to get correct estimates of the opponent location [3], [4], [6], [12] and 2) to improve policies for the S&T agents by leveraging the estimation from the filtering module [1], [2], [5], [8]- [10], [13] The classic Kalman-based filters have been widely used in this field. Interacting Mutiple Model (IMM) and Probability Hypothesis Density (PHD) filters are used to solve the multiple dynamic patterns and the unknown varying number of agents by Kalman model merging.…”

“…More recent work mainly focuses on searching and tracking strategies to improve searching efficiency and data collection [1], [2], [5], [7], [9], [10], [13], [15]. One type of strategy is to use heuristic policies like spirals, lawn-movers [9], and interceptions [13].…”

“…One type of strategy is to use heuristic policies like spirals, lawn-movers [9], and interceptions [13]. However, they either assume they know the ground truth location [13], have a large field of view [1], or search non-adversarial targets that will not evade [2], [10]. In this paper, we utilize learning-based policies [2], [5], [10] to outperform the hand-coded heuristic policies.…”

“…Adversarial S&T systems have been developed to address these challenges and provide more robust and accurate tracking capabilities [1]. These systems often require parallel and hierarchical architectures, including probability map construction [2], detection-object association [3]- [5], target location filtering [3], [4], [6], and searching strategy optimization [2], [7]. The appropriate design, organization, and coordination of these *This work supported in part by Office of Naval Research (ONR) under grant numbers N00014-19-1-2076, N00014-22-1-2834, and N00173-21-1-G009, the National Science Foundation under grant CNS-2219755, and MIT Lincoln Laboratory grant number 7000437192.…”

“…Multi-Agent Reinforcement Learning has shown promising results in learning agent policies that outperform handtuned expert heuristics [7]. However, applying MARL in an adversarial domain poses challenges due to sparse observations and rewards.…”