Scanner

Abstract

A growing number of visual computing applications depend on the analysis of large video collections. The challenge is that scaling applications to operate on these datasets requires efficient systems for pixel data access and parallel processing across large numbers of machines. Few programmers have the capability to operate efficiently at these scales, limiting the field's ability to explore new applications that leverage big video data. In response, we have created Scanner, a system for productive and efficient video analysis at scale. Scanner organizes video collections as tables in a data store optimized for sampling frames from compressed video, and executes pixel processing computations, expressed as dataflow graphs, on these frames. Scanner schedules video analysis applications expressed using these abstractions onto heterogeneous throughput computing hardware, such as multi-core CPUs, GPUs, and media processing ASICs, for high-throughput pixel processing. We demonstrate the productivity of Scanner by authoring a variety of video processing applications including the synthesis of stereo VR video streams from multi-camera rigs, markerless 3D human pose reconstruction from video, and data-mining big video datasets such as hundreds of feature-length films or over 70,000 hours of TV news. These applications achieve near-expert performance on a single machine and scale efficiently to hundreds of machines, enabling formerly long-running big video data analysis tasks to be carried out in minutes to hours.

VStore

“…Targeting NN training, NVIDIA's Video Loader [14] (a wrapper over NVDEC and FFmpeg) optimizes random loads of encoded video frames. To support video analytics at scale, Scanner [51] organizes video collections and raster data as tables in a data store and executes costly pixel-level computations in parallel. All these systems are short on controlling visual data formats according to analytics.…”

Adolescent Violence through Technology in India

“…Facebook created their photo management application as an ad-hoc solution of Tao [31] (data store), Haystack [2] (object store), f4 [21] (blob storage system), and MySQL [22] (metadata storage). Many solutions have been developed recently to cater to video data specifically [1,4,7,13,14,17,25]. Scanner [25] is an open-source system designed for efficient video processing.…”

“…Many solutions have been developed recently to cater to video data specifically [1,4,7,13,14,17,25]. Scanner [25] is an open-source system designed for efficient video processing. It provides tools and APIs to perform a variety of video processing operations such as object detection, face detection, pose detection, etc.…”

“…Moreover, it follows a synchronous model to execute an operation pipeline that could lead to execution delays (Details in Section 3). Other existing video data management systems [7,14,25] are also synchronous or use a restricted parallelization approach. Furthermore, these solutions do not support operation execution at a remote server and require an ad-hoc setup to achieve remote execution.…”

Accelerating Aggregation Queries on Unstructured Streams of Data

“…Prior work has leveraged deep neural networks (DNNs) to execute queries over large datasets of unstructured data [4,17,20,21,24,25,30,40,47,48]. For example, the social scientist might use a BERT model [14] to execute the following query:…”