Closed-Loop Bayesian Semantic Data Fusion for Collaborative Human-Autonomy Target Search

Abstract

In search applications, autonomous unmanned vehicles must be able to efficiently reacquire and localize mobile targets that can remain out of view for long periods of time in large spaces. As such, all available information sources must be actively leveraged -including imprecise but readily available semantic observations provided by humans. To achieve this, this work develops and validates a novel collaborative human-machine sensing solution for dynamic target search. Our approach uses continuous partially observable Markov decision process (CPOMDP) planning to generate vehicle trajectories that optimally exploit imperfect detection data from onboard sensors, as well as semantic natural language observations that can be specifically requested from human sensors. The key innovation is a scalable hierarchical Gaussian mixture model formulation for efficiently solving CPOMDPs with semantic observations in continuous dynamic state spaces. The approach is demonstrated and validated with a real human-robot team engaged in dynamic indoor target search and capture scenarios on a custom testbed.

Optimal Continuous State POMDP Planning With Semantic Observations: A Variational Approach

“…In particular, it is desirable to obtain bounds on the accuracy of the Kmeans hybrid GM condensation method, as well as possible lower bounds on the value function via the VB inference approximation. Finally, building on previous work in [3], [27] and ongoing work in [7], the CPOMDP framework developed here will be leveraged for cooperative human-robot target search and tracking applications, so that semantic 'human sensor data' from natural language inputs can be combined with optimal robotic sensing and motion planning in hardware for tightly integrated human-robot teaming.…”

“…The approach described in this paper can be used to solve for combined motion planning and human querying policies offline, thus avoiding high computational cost and achieving tighter integration of planning and sensing with complex uncertainties. Application to the full semantic active sensing problem with human-robot teams is not treated in detail here, but has been implemented and examined in related work [7].…”

“…This paper also provides additional comparisons to state of the art POMDP policy approximations and assesses the robustness of the VB GM policy approximation to model errors. Though beyond the scope of this paper, the methods developed here have also been successfully adapted and deployed in hardware for collaborative human-robot target search and interception [7].…”

An Adaptive Multi-Homogeneous Sensor Weight Calculation Method for Body Sensor Networks

“…Different fusion methods promote the diversity of BSNs application. Recent studies have focused on target recognition [17]- [20], behavior recognition [21]- [23], emotion recognition [24], [25], disease diagnosis [26]- [29], physical rehabilitation [30]- [32] and other aspects. Shu et al [24] presented a comprehensive review on physiological signalbased emotion recognition and conduct a summary and comparison of the recent studies.…”

“…In order to compare with other fusion algorithms more accurately, the performance evaluation method used in [40] is cited, as shown in Eq. (20). The average of absolute difference can reflect the average deviation degree of two sets of sequences.…”