Low-Latency Proactive Continuous Vision

Abstract

Continuous vision is the cornerstone of a diverse range of intelligent applications found on emerging computing platforms such as autonomous machines and Augmented Reality glasses. A critical issue in today's continuous vision systems is their long end-to-end frame latency, which significantly impacts the system agility and user experience. We find that the long latency is fundamentally caused by the serialized execution model of today's continuous vision pipeline, whose key stages, including sensing, imaging, and vision computations, execute sequentially, leading to long frame latency. This paper seeks to reduce the end-to-end latency of continuous vision tasks. Our key idea is a new proactive vision execution model that breaks the sequential execution of the vision pipeline. Specifically, we propose to allow the pipeline front-end (sensing and imaging) to predict future frames; the pipeline back-end (vision algorithms) then predictively operates on the future frames to reduce frame latency. While the proactive execution model is generally applicable to any vision systems, we demonstrate its effectiveness using an implementation on resource-constrained mobile Systemson-a-chips (SoC). Our system, PVF, incorporates two techniques to overcome key challenges that arise in deploying proactive vision in mobile systems: it enables multiple outstanding speculative frames by exploiting the hardware heterogeneities in mobile SoCs; it reduces the energy overhead of prediction by exploiting the error-resilient nature of vision algorithms. We show that PVF reduces the frame latency by up to 92% under the same energy. CCS Concepts • Computer systems organization → Heterogeneous (hybrid) systems; Special purpose systems.

P2M-DeTrack: Processing-in-Pixel-in-Memory for Energy-efficient and Real-Time Multi-Object Detection and Tracking

“…We use the per-pixel communication delay from [9] to estimate the delay required to send the output activation maps from sensor to SoC (T com ). While T sens is primarily governed by the sensor read delay, T adc is associated with the total number of ADC clock cycles required to generate the first 3 The pixel array energy is equal to the image read-out energy for the baseline models and in-pixel convolution energy for custom models.…”

VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

“…Deep learning (DL) models have achieved great success in the various domains including vision [29,36,49,50,52,53], natural language processing [15,24], and even graph learning [62,67]. To meet the need of rising computation power of DL models, computer architects have proposed various hardware designs including general-purpose hardware [45] and domain-specific architectures [7,10,18,19,26,27,32,58,61,68] for accelerating deep learning models for their superior energy efficiency.…”

VELTAIR: Towards High-Performance Multi-tenant Deep Learning Services via Adaptive Compilation and Scheduling

“…Deep learning (DL) models have achieved great success in the various domains including vision [1,2,3,4,5,6], natural language processing [7,8], and even graph learning [9,10]. To meet the need of rising computation power of DL models, computer architects have proposed various hardware designs including general-purpose hardware [11] and domain-specific architectures [12,13,14,15,16,17,18,19,20,21] for accelerating deep learning models for their superior energy efficiency.…”