STAC: a comprehensive sensor fusion model for scene characterization

Abstract

We are interested in data fusion strategies for Intelligence, Surveillance, and Reconnaissance (ISR) missions. Advances in theory, algorithms, and computational power have made it possible to extract rich semantic information from a wide variety of sensors, but these advances have raised new challenges in fusing the data. For example, in developing fusion algorithms for moving target identification (MTI) applications, what is the best way to combine image data having different temporal frequencies, and how should we introduce contextual information acquired from monitoring cell phones or from human intelligence? In addressing these questions we have found that existing data fusion models do not readily facilitate comparison of fusion algorithms performing such complex information extraction, so we developed a new model that does. Here, we present the Spatial, Temporal, Algorithm, and Cognition (STAC) model. STAC allows for describing the progression of multi-sensor raw data through increasing levels of abstraction, and provides a way to easily compare fusion strategies. It provides for unambiguous description of how multi-sensor data are combined, the computational algorithms being used, and how scene understanding is ultimately achieved. In this paper, we describe and illustrate the STAC model, and compare it to other existing models. INTRODUCTIONSensors, algorithms and computational power continue to evolve rapidly. Beyond physical sensors such as cameras, lidars, and spectrometers, cell phones and social media generate contextual information of growing importance in surveillance applications. However, the sensor fusion models used to describe and compare processing architectures have been comparatively stagnant. Important models such as the JDL (Joint Directors of Laboratories) model [1, 2] have been revised and updated over time, but have not been sufficiently generalized to reflect the current state of the art in sensors or algorithms. The limitations of existing fusion models become especially clear when working in scene characterization and moving target identification (MTI) applications. These applications require fine-grained information about tracked objects including detailed attributes about them. These attributes may be inferred from multi-sensor sensor data, relationships between entities, behavioral patterns recurring through time, and inference of activities and intents of these entities.In this paper we introduce a new fusion model with the goal of addressing these limitations: the Spatial, Temporal, Algorithm, and Cognition (STAC) model. Like other models, STAC provides an organizational basis for describing the extraction of multiple information streams from multi-source data, and the multi-level fusion of these information streams leading to cognition. However, STAC is novel in that it provides for the development of cognition over time.We believe this temporal dimension to be increasingly important for change detection and MTI applications because the source data are often acquired over diffe...

ISR Data Processing in Military Operations

“…The data fusion strategies for ISR missions, similar to previous paper, is presented in [17]. Extraction of sematic information from various set of sensors and its processing is based on new AI algorithms and computational power.…”