Server-Driven Video Streaming for Deep Learning Inference

Abstract

Video streaming is crucial for AI applications that gather videos from sources to servers for inference by deep neural nets (DNNs). Unlike traditional video streaming that optimizes visual quality, this new type of video streaming permits aggressive compression/pruning of pixels not relevant to achieving high DNN inference accuracy. However, much of this potential is left unrealized, because current video streaming protocols are driven by the video source (camera) where the compute is rather limited. We advocate that the video streaming protocol should be driven by real-time feedback from the server-side DNN. Our insight is twofold: (1) server-side DNN has more context about the pixels that maximize its inference accuracy; and (2) the DNN's output contains rich information useful to guide video streaming. We present DDS (DNN-Driven Streaming), a concrete design of this approach. DDS continuously sends a low-quality video stream to the server; the server runs the DNN to determine where to re-send with higher quality to increase the inference accuracy. We find that compared to several recent baselines on multiple video genres and vision tasks, DDS maintains higher accuracy while reducing bandwidth usage by upto 59% or improves accuracy by upto 9% with no additional bandwidth usage.

Catching the Moment With LoL$^+$ in Twitch-Like Low-Latency Live Streaming Platforms

“…Thereafter, for each cluster, it trains a hidden Markov model (HMM) to estimate the corresponding bandwidth. A number of other data-driven machine learning solutions have also emerged: AMP [4], CFA [18], PREM [39], LiveNAS [22], Pytheas [20] and DDS [12].…”

Machine Learning for Multimedia Communications

“…This calls for alternative video streaming systems that forecast users’ requests, exploit correlation among video consumption patterns of the users and possible association of users with an access point or base station. As forecasting is central in these systems, machine learning methods have been proposed for both 360-degree video [ 127 , 128 , 129 , 130 , 131 ] and regular video [ 132 , 133 , 134 ].…”

“…A CNN is used to extract image features and an RNN to capture temporal features. A sender-driven video streaming system is employed in [ 133 ] for improved inference. A deep neural network is used to extract the areas of interest from a low-quality video.…”

“…For such problems, the difficulty naturally comes when several processing tasks are considered, or even worse, when the processing task is unknown at the encoding stage. Some works have been proposed for image compression [ 208 , 209 ] or video streaming protocol redefinition [ 133 ]. In [ 210 , 211 ], features-based compression algorithms aim at maximizing quality perceived both by humans and machines.…”

A Review of Efficient Real-Time Decision Making in the Internet of Things

“…Li et al [76], on-camera filtering is performed for efficient realtime video analytics. In [77], an IoT camera analyzes the traffic flow using a low-resolution image and the edge server also analyzes the image, identifies an important part of the image (if any) in terms of data analytics, and requests an important part in high resolution from the device. IoT devices in a smart building can transfer their sensor readings to the IoT gateway on the same floor for efficient HVAC (Heating, Ventilation, and Air Conditioning).…”