Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking

Abstract

The ability to detect and track objects in the visual world is a crucial skill for any intelligent agent, as it is a necessary precursor to any object-level reasoning process. Moreover, it is important that agents learn to track objects without supervision (i.e. without access to annotated training videos) since this will allow agents to begin operating in new environments with minimal human assistance. The task of learning to discover and track objects in videos, which we call unsupervised object tracking, has grown in prominence in recent years; however, most architectures that address it still struggle to deal with large scenes containing many objects. In the current work, we propose an architecture that scales well to the large-scene, many-object setting by employing spatially invariant computations (convolutions and spatial attention) and representations (a spatially local object specification scheme). In a series of experiments, we demonstrate a number of attractive features of our architecture; most notably, that it outperforms competing methods at tracking objects in cluttered scenes with many objects, and that it can generalize well to videos that are larger and/or contain more objects than videos encountered during training.

Active Vision for Robot Manipulators Using the Free Energy Principle

“…In SQ-AIR, this work is extended to sequences of images, and a discovery and propagation mechanism was introduced to track objects through different frames (Kosiorek et al, 2018 ). These have been extended to better handle physical interactions (Kossen et al, 2020 ) or be more scalable (Crawford and Pineau, 2020 ; Jiang et al, 2020 ). These extensions have also been combined by Lin et al ( 2020 ).…”

Robust Unsupervised Multi-Object Tracking In Noisy Environments

“…Spatially invariant attendinfer-repeat (SPAIR) models [7] used CNN feature extractors and a local spatial object specification scheme to conduct UMOT video processing with variational inference. Building upon the SPAIR method, spatially invariant label-free object tracking (SILOT) [4] incorporated VAE based architecture with competitive MOT accuracy on MNIST-MOT and Atari video games. However, previous methods and benchmark models do not provide experimental studies on the effect of background noise on tracker performance.…”

“…As shown in Figure 3, we select two baseline video datasets from the previous UMOT studies: (a) MNIST-MOT [3,7] containing 2M training frames and 25k validation frames, where each frame is of size 128 × 128 × 1 and (b) Atari gaming video on Space-Invader (Atari-SI) [4,7] containing 128k training frames and 1k validation frames, where each frame is converted to gray-scale with a input size of 210 × 160 × 1 for testing.…”

“…• Spatially invariant label-free object tracking (SILOT) [4]: a model built upon VAE for frame-wise reconstruction and using YOLO-based CNN feature extraction for representation learning.…”

“…• Evaluation of the proposed method on MNIST-MOT [3], Atari gaming video [4], and two newly created UMOT datasets with complex patterns: (1) Kuzushiji-MNIST MOT derived from [5] and (2) Fashion-MNIST MOT derived from [6].…”