Deep Neural Network–based Enhancement for Image and Video Streaming Systems: A Survey and Future Directions

Abstract

Internet-enabled smartphones and ultra-wide displays are transforming a variety of visual apps spanning from on-demand movies and 360° videos to video-conferencing and live streaming. However, robustly delivering visual content under fluctuating networking conditions on devices of diverse capabilities remains an open problem. In recent years, advances in the field of deep learning on tasks such as super-resolution and image enhancement have led to unprecedented performance in generating high-quality images from low-quality ones, a process we refer to as neural enhancement. In this article, we survey state-of-the-art content delivery systems that employ neural enhancement as a key component in achieving both fast response time and high visual quality. We first present the components and architecture of existing content delivery systems, highlighting their challenges and motivating the use of neural enhancement models as a countermeasure. We then cover the deployment challenges of these models and analyze existing systems and their design decisions in efficiently overcoming these technical challenges. Additionally, we underline the key trends and common approaches across systems that target diverse use-cases. Finally, we present promising future directions based on the latest insights from deep learning research to further boost the quality of experience of content delivery systems.

OODIn: An Optimised On-Device Inference Framework for Heterogeneous Mobile Devices

“…At the same time, AI applications come with various performance needs. These span from latency-critical apps, such as smart cameras and Augmented Reality (AR) [13], to the throughputoriented demands of high-resolution video understanding [3] and image enhancement [14]. Finally, DL models tend to have vastly different computational costs.…”

Use of scanning devices for object 3D reconstruction by photogrammetry and visualization in virtual reality

“…These models can serve as a basis for 3D printing [3], [4] or be used in augmented reality and virtual reality (AR/VR) [5]. The goal of the captured image is not only its high quality but also the short processing time of this image and its sharing or stream [6]. There are several methods for creating 3D models.…”

“…Photogrammetry works on the principle of using 2D image data, most often in the form of photographs. Therefore, classic compact cameras, mobile cameras, 3D or 360 cameras, laser 3D sceners, drones, or tablets with LiDar technologies, which are now available to the general public, are used for this method [6], [9], [10], [20]. The quality of the scanned image depends on the resolution, color gamut, lighting conditions, and attributes are affect the demand for additional outputs [12], [20], [21].…”

fakeWeather: Adversarial Attacks for Deep Neural Networks Emulating Weather Conditions on the Camera Lens of Autonomous Systems

“…In the last decade, Deep Neural Networks (DNNs) have lifted groundbreaking advancements in several fields, including object recognition [1], autonomous systems [2], video, image, and signal processing [3], and achieving the human-level or even more classification accuracy for certain tasks [4]. However, despite their great success [5] [6], DNNs have been proven to be vulnerable to adversarial attacks, which undermine their security since they maliciously subvert the DNN predictions [7].…”