A Decentralized Partially Observable Decision Model for Recognizing the Multiagent Goal in Simulation Systems

Abstract

Multiagent goal recognition is important in many simulation systems. Many of the existing modeling methods need detailed domain knowledge of agents’ cooperative behaviors and a training dataset to estimate policies. To solve these problems, we propose a novel decentralized partially observable decision model (Dec-POMDM), which models cooperative behaviors by joint policies. In this compact way, we only focus on the distribution of joint policies. Additionally, a model-free algorithm, cooperative colearning based on Sarsa, is exploited to estimate agents’ policies under the assumption of rationality, which makes the training dataset unnecessary. In the inference, considering that the Dec-POMDM is discrete and its state space is large, we implement a marginal filter (MF) under the framework of the Dec-POMDM, where the initial world states and results of actions are uncertain. In the experiments, a new scenario is designed based on the standard predator-prey problem: we increase the number of preys, and our aim is to recognize the real target of predators. Experiment results show that (a) our method recognizes goals well even when they change dynamically; (b) the Dec-POMDM outperforms supervised trained HMMs in terms of precision, recall, and F-measure; and (c) the MF infers goals more efficiently than the particle filter under the framework of the Dec-POMDM.

A Decentralized Partially Observable Markov Decision Model with Action Duration for Goal Recognition in Real Time Strategy Games

“…The CxHSMM modifies HMM in two aspects: on one hand, it is a special DBN representation of two-layer HMM, and it also has termination variables; on the other hand, it used Coxian distribution to model the duration of primitive actions explicitly. Besides, Yue et al [9] proposed a SMDM (semi-Markov Decision Model) based on AHMM, which not only has hierarchical structure, but also models the time duration. Similar methods also include Semi-Markov CRF (SMCRF) [29] and Hierarchical Semi-Markov CRF (HSCRF) [30].…”

“…Ramirez and Geffner also solved the inference problem even when observations are incomplete. Besides, Yue et al [9] also proposed a Dec-POMDM model based on Dec-POMDP in recognizing multiagent goal recognition. Its model, however, does not consider situations when agents are having durative actions in RTS games.…”

“…Several modifications based on the MDP framework have a finer formalization towards more complex scenarios. Among these models, the Dec-POMDM (decentralized partially observable Markov decision model) [9] is a MDP-based method focusing on solving multiagent goal recognition problem. Though having all details of cooperation embedded in team's joint policy, Dec-POMDM is only concerned about actions starting and terminating within one time step.…”

“…The Dec-POMDM-T models the joint goal, the actions, and the world states hierarchically. Compared to works in [9,11], Dec-POMDM-T explicitly models the time duration for primitive actions, indicating whether actions are terminated or not. In Dec-POMDM-T, the multiagent joint goal recognition consists of three components: (a) formalization of behaviors, the environment, and the observation for organizers; (b) model parameter estimation through learning or other methods; and (c) goal inference from observations:…”

“…Like the modified predator-prey problem presented in [9], the scenario in this paper also has more than one prey and predator. The predators first establish joint pursuing target or goal, which would be changed halfway, before capturing it.…”