Accelerating Machine Learning Inference with Probabilistic Predicates

Lu, Yao; Chowdhery, Aakanksha; Kandula, Srikanth; Chaudhuri, Surajit

doi:10.1145/3183713.3183751

Cited by 89 publications

(107 citation statements)

References 34 publications

Supporting

Mentioning

107

Contrasting

Order By: Relevance

“…However, while existing work on video query optimization support queries involving aggregation over entire videos, they focus on predicates that can be evaluated over individual frames. Although probabilistic predicates [16] extends specialized classifiers for predicates over sequences of frames, we will show in our evaluation that this approach is not effective for most object track queries. Additionally, prior optimization approaches do not accelerate queries where instances of the object type of interest appear in every frame.…”

Section: Related Workmentioning

confidence: 99%

“…PP implements the Deep Neural Network (DNN) classifier in probabilistic predicates [16]. We train the DNN to classify whether 5-second segments of video contain any object instances that satisfy the query predicate.…”

Section: Baselinesmentioning

confidence: 99%

“…Recently, several video query processing engines have been proposed to accelerate the execution of video analytics queries by addressing the GPU-intensiveness of object detection. Broadly, these engines, which include NoScope [13], probabilistic predicates [16], BlazeIt [12], and SVQ [21], train lightweight, specialized machine learning models to approximate the result of a predicate over individual video frames or sequences of frames. If the lightweight model is confident that a segment of video does not contain any object instances that satisfy the predicate, the query processor skips expensive object detector execution over the video segment.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

MIRIS: Fast Object Track Queries in Video

Bastani

Balasingam

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Video databases that enable queries with object-track predicates are useful in many applications. Such queries include selecting objects that move from one region of the camera frame to another (e.g., finding cars that turn right through a junction) and selecting objects with certain speeds (e.g., finding animals that stop to drink water from a lake). Processing such predicates efficiently is challenging because they involve the movement of an object over several video frames. We propose a novel query-driven tracking approach that integrates query processing with object tracking to efficiently process object track queries and address the computational complexity of object detection methods. By processing video at low framerates when possible, but increasing the framerate when needed to ensure high-accuracy on a query, our approach substantially speeds up query execution. We have implemented query-driven tracking in MIRIS, a video query processor, and compare MIRIS against four baselines on a diverse dataset consisting of five sources of video and nine distinct queries. We find that, at the same accuracy, MIRIS accelerates video query processing by 9x on average over the IOU tracker, an overlap-based tracking-by-detection method used in existing video database systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

MIRIS: Fast Object Track Queries in Video

Bastani

Balasingam

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…We consider three configurations: (1) ODIN with specialized models and no filters, (2) ODIN-PP with specialized models and unspecialized filter [23], and (3) ODIN-FILTER with specialized models and specialized filters. The results are shown in Table 6.…”

Section: Aggregation Querymentioning

confidence: 99%

Odin

Suprem

Arulraj

et al. 2020

Proc. VLDB Endow.

View full text Add to dashboard Cite

Recent advances in computer vision have led to a resurgence of interest in visual data analytics. Researchers are developing systems for effectively and efficiently analyzing visual data at scale. A significant challenge that these systems encounter lies in the drift in real-world visual data. For instance, a model for self-driving vehicles that is not trained on images containing snow does not work well when it encounters them in practice. This drift phenomenon limits the accuracy of models employed for visual data analytics. In this paper, we present a visual data analytics system, called ODIN, that automatically detects and recovers from drift. ODIN uses adversarial autoencoders to learn the distribution of highdimensional images. We present an unsupervised algorithm for detecting drift by comparing the distributions of the given data against that of previously seen data. When ODIN detects drift, it invokes a drift recovery algorithm to deploy specialized models tailored towards the novel data points. These specialized models outperform their non-specialized counterpart on accuracy, performance, and memory footprint. Lastly, we present a model selection algorithm for picking an ensemble of best-fit specialized models to process a given input. We evaluate the efficacy and efficiency of ODIN on high-resolution dashboard camera videos captured under diverse environments from the Berkeley DeepDrive dataset. We demonstrate that ODIN's models deliver 6× higher throughput, 2× higher accuracy, and 6× smaller memory footprint compared to a baseline system without automated drift detection and recovery.

show abstract

“…Database researchers have been working on exploiting the optimization techniques from database systems like batch processing, indexing, caching, etc. for optimizing machine learning systems or tasks [18,13,7,3,16]. Clipper [7] proposes a general system for machine learning inference, which batches the messages received from users and feed them into the model for higher throughput; MacroBase [3] accelerates the analytics of fast data streams by applying the machine learning models only over sampled data.…”

Section: Optimization For Machine Learningmentioning

confidence: 99%

A distributed system for large-scale n-gram language models at Tencent

Long¹,

Wang

Deng³

et al. 2019

Proc. VLDB Endow.

View full text Add to dashboard Cite

n-gram language models are widely used in language processing applications, e.g., automatic speech recognition, for ranking the candidate word sequences generated from the generator model, e.g., the acoustic model. Large n-gram models typically give good ranking results; however, they require a huge amount of memory storage. While distributing the model across multiple nodes resolves the memory issue, it nonetheless incurs a great network communication overhead and introduces a different bottleneck. In this paper, we present our distributed system developed at Tencent with novel optimization techniques for reducing the network overhead, including distributed indexing, batching and caching. They reduce the network requests and accelerate the operation on each single node. We also propose a cascade fault-tolerance mechanism which adaptively switches to small n-gram models depending on the severity of the failure. Experimental study on 9 automatic speech recognition (ASR) datasets confirms that our distributed system scales to large models efficiently, effectively and robustly. We have successfully deployed it for Tencent's WeChat ASR with the peak network traffic at the scale of 100 millions of messages per minute.

show abstract

Accelerating Machine Learning Inference with Probabilistic Predicates

Cited by 89 publications

References 34 publications

MIRIS: Fast Object Track Queries in Video

MIRIS: Fast Object Track Queries in Video

Odin

A distributed system for large-scale n-gram language models at Tencent

Contact Info

Product

Resources

About