Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels

Abstract

Many real-world video analysis applications require the ability to identify domain-specific events in video, such as interviews and commercials in TV news broadcasts, or action sequences in film. Unfortunately, pre-trained models to detect all the events of interest in video may not exist, and training new models from scratch can be costly and labor-intensive. In this paper, we explore the utility of specifying new events in video in a more traditional manner: by writing queries that compose outputs of existing, pre-trained models. To write these queries, we have developed REKALL, a library that exposes a data model and programming model for compositional video event specification. REKALL represents video annotations from different sources (object detectors, transcripts, etc.) as spatiotemporal labels associated with continuous volumes of spacetime in a video, and provides operators for composing labels into queries that model new video events. We demonstrate the use of REKALL in analyzing video from cable TV news broadcasts, films, static-camera vehicular video streams, and commercial autonomous vehicle logs. In these efforts, domain experts were able to quickly (in a few hours to a day) author queries that enabled the accurate detection of new events (on par with, and in some cases much more accurate than, learned approaches) and to rapidly retrieve video clips for human-in-the-loop tasks such as video content curation and training data curation. Finally, in a user study, novice users of REKALL were able to author queries to retrieve new events in video given just one hour of query development time.

Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods

“…Video Analysis Tasks. We use video analysis as another driving task: video data is large and expensive to label, and modeling temporal dependencies is important for quality but introduces significant slowdowns in label model parameter recovery (Sala et al, 2019) a corpus of TV news, respectively (Fu et al, 2019;Int, 2018).…”

Leveraging organizational resources to adapt models to new data modalities

“…This is challenging in our setting for two reasons. First, generating LFs over unstructured data modalities (such as image and video) is an active area of research [27,54,65,67,76]. In contrast, defining predicates is straightforward over structured feature spaces such as text (e.g., string or pattern matchers), quantitative data (e.g., thresholds), or categorical data (e.g., checking presence of a topic) variables.…”

Multi-Resolution Weak Supervision for Sequential Data

“…Interview Detection (Interview): We use the dataset from [14] and use the dev/test splits from that work. We additionally use an additional 57 hours of unlabelled data as the train split.…”

“…Consider training a deep learning model to detect interviews in TV news videos [14]. As shown in Figure 1, supervision sources to generate training labels can draw on indirect signals from closed caption transcripts (per-scene), bounding box movement between frames (per-window), and pixels in the background of each frame (per-frame).…”