Adaptive Streaming Perception using Deep Reinforcement Learning

Ghosh, Anurag; Nambi, Akshay; Balloli, Vaibhav; Singh, Aditya; Ganu, Tanuja

doi:10.48550/arxiv.2106.05665

Cited by 3 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Likewise, if the edge device experiences thermal throttling or is constrained by power consumption, then lowering edge detection frequency is necessary (say for battery-operated drones). Concurrent work [12] has shown the feasibility of learning configurations for live streaming applications via Reinforcement Learning.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Streaming Video Analytics On The Edge With Asynchronous Cloud Support

Ghosh¹,

Iyengar²,

Rathore³

et al. 2022

Preprint

View full text Add to dashboard Cite

Emerging Internet of Things (IoT) and mobile computing applications are expected to support latency-sensitive deep neural network (DNN) workloads. To realize this vision, the Internet is evolving towards an edge-computing architecture, where computing infrastructure is located closer to the end device to help achieve low latency. However, edge computing may have limited resources compared to cloud environments and thus, cannot run large DNN models that often have high accuracy.In this work, we develop REACT, a framework that leverages cloud resources to execute large DNN models with higher accuracy to improve the accuracy of models running on edge devices. To do so, we propose a novel edge-cloud fusion algorithm that fuses edge and cloud predictions, achieving low latency and high accuracy. We extensively evaluate our approach and show that our approach can significantly improve the accuracy compared to baseline approaches. We focus specifically on object detection in videos (applicable in many video analytics scenarios) and show that the fused edge-cloud predictions can outperform the accuracy of edge-only and cloud-only scenarios by as much as 50%. We also show that REACT can achieve good performance across tradeoff points by choosing a wide range of system parameters to satisfy use-case specific constraints, such as limited network bandwidth or GPU cycles.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Streaming Video Analytics On The Edge With Asynchronous Cloud Support

Ghosh¹,

Iyengar²,

Rathore³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In order to improve perception performance, previous works have explored the concept of streaming perception, which utilizes temporal information. For example, Li et al [37] introduced a benchmark for image detection algorithms and proposed a method based on Kalman filtering [38] and reinforcement learning [39] to mitigate latency. Han et al [40] developed an efficient streaming detector for LiDAR-based 3D detection tasks, accurately predicting future frames.…”

Section: Streaming Perceptionmentioning

confidence: 99%

Benchmarking Perception to Streaming Inputs in Vision-Centric Autonomous Driving

Jin,

Ding,

Yang

et al. 2023

Mathematics

View full text Add to dashboard Cite

In recent years, vision-centric perception has played a crucial role in autonomous driving tasks, encompassing functions such as 3D detection, map construction, and motion forecasting. However, the deployment of vision-centric approaches in practical scenarios is hindered by substantial latency, often deviating significantly from the outcomes achieved through offline training. This disparity arises from the fact that conventional benchmarks for autonomous driving perception predominantly conduct offline evaluations, thereby largely overlooking the latency concerns prevalent in real-world deployment. Although a few benchmarks have been proposed to address this limitation by introducing effective evaluation methods for online perception, they do not adequately consider the intricacies introduced by the complexity of input information streams. To address this gap, we propose the Autonomous driving Streaming I/O (ASIO) benchmark, aiming to assess the streaming input characteristics and online performance of vision-centric perception in autonomous driving. To facilitate this evaluation across diverse streaming inputs, we initially establish a dataset based on the CARLA Leaderboard. In alignment with real-world deployment considerations, we further develop evaluation metrics based on information complexity specifically tailored for streaming inputs and streaming performance. Experimental results indicate significant variations in model performance and ranking under different major camera deployments, underscoring the necessity of thoroughly accounting for the influences of model latency and streaming input characteristics during real-world deployment. To enhance streaming performance consistently across distinct streaming input features, we introduce a backbone switcher based on the identified streaming input characteristics. Experimental validation demonstrates its efficacy in perpetually improving streaming performance across varying streaming input features.

show abstract

“…[26] proposes a meta-detector to alleviate this problem by employing Kalman filter [25], decisiontheoretic scheduling, and asynchronous tracking [1]. [16] lists several factors (e.g., input scales, switchability of detectors, and scene aggregation.) and designs a reinforcement learning-based agent to learn a better combination for a better trade-off.…”

Section: Related Workmentioning

confidence: 99%

“…Further, [26] proposes a meta-detector named Streamer that can incorporate any detector with decisiontheoretic scheduling, asynchronous tracking, and future forecasting to recover much of the performance drop. Following this work, Adaptive streamer [16] adopts numerous approximate executions based on deep reinforcement learning to learn a better trade-off online. These works focus on searching for a better trade-off policy between speed and accuracy for some existing detectors, while a novel streaming perception model design is not well studied.…”

Section: Introductionmentioning

confidence: 99%

Real-time Object Detection for Streaming Perception

Yang¹,

Liu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Autonomous driving requires the model to perceive the environment and (re)act within a low latency for safety. While past works ignore the inevitable changes in the environment after processing, streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. In this paper, instead of searching trade-offs between accuracy and speed like previous works, we point out that endowing real-time models with the ability to predict the future is the key to dealing with this problem. We build a simple and effective framework for streaming perception. It equips a novel Dual-Flow Perception module (DFP), which includes dynamic and static flows to capture the moving trend and basic detection feature for streaming prediction. Further, we introduce a Trend-Aware Loss (TAL) combined with a trend factor to generate adaptive weights for objects with different moving speeds. Our simple method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline, validating its effectiveness. Our code will be made available at https: //github.com/yancie-yjr/StreamYOLO.

show abstract

Adaptive Streaming Perception using Deep Reinforcement Learning

Cited by 3 publications

References 24 publications

Streaming Video Analytics On The Edge With Asynchronous Cloud Support

Streaming Video Analytics On The Edge With Asynchronous Cloud Support

Benchmarking Perception to Streaming Inputs in Vision-Centric Autonomous Driving

Real-time Object Detection for Streaming Perception

Contact Info

Product

Resources

About