Apparent resolution enhancement for motion videos

Abstract

In this work we increase the apparent resolution of videos when viewed on a high-refresh rate display by making use of perceptual properties of the visual system. We achieve this enhancement by exploiting the viewer's natural tendency to track moving objects in videos which causes the screen pixels to be projected at different sub-pixel offsets onto the retina. We estimate the eye motion using optical flow and use it to compute multiple low-resolution frames for each input frame. By watching these new frames at a high frame-rate, the viewer's eyes integrate them over time and merges them into a single perceived frame with a denser pixel layout. In this work we also advance the existing approaches for resolution enhancement in the following ways. We combine current display resolution enhancement with super-resolution methods to enhance input videos that are at the display resolution. We derive a new perceived video model that accounts for actual camera sensor and display pixel shapes in order to achieve optimal enhancement. We analyze the degeneracies that certain motion velocities introduce to super-resolution and resolution enhancement, and offer algorithmic solutions for handling these scenarios as well as other difficulties that arise when dealing with the optical flow of natural videos.A user study finds that our approach achieves a noticeable increase in the apparent resolution for videos even when viewed on regular hardware (60Hz), and further enhances resolution when viewed on higher refresh rate displays (120Hz).

A Total Variation Approach for Customizing Imagery to Improve Visual Acuity

“…Within perception [Cornsweet 1970], phenomena such as edge enhancement, opponent-color theory, psychophysical reading aspects (e.g., [Legge et al 1985]), and numerous so-called visual illusions are exploited (e.g., [Wandell 1995;Pizlo et al 2007;Pizlo 2010]). Some approaches combine perceptually-based and ophthalmologically-based techniques for improving image viewing (e.g., [Peli and Woods 2009]) and for improving resolution in videos and animations (e.g., [Didyk et al 2010;Berthouzoz and Fattal 2012a;Berthouzoz and Fattal 2012b;Templin et al 2013;Stengel et al 2013]). For instance, the Cornsweet illusion has been exploited to provide enhanced local contrast [Ritschel et al 2008] and to provide a perceptually-based metric for achieving compelling and efficient stereo disparity [Didyk et al 2011].…”

Optimizing Apparent Display Resolution Enhancement for Arbitrary Videos

“…While Didyk et al [3] demonstrated the applicability of their approach only for linear motion, Templin et al [5] transformed it into an optimization problem which could be used for arbitrary motion in animations. Fattal et al combine this approach with a super-resolution PREVIOUS MODEL method to spatially enhance input videos that are at the display resolution [6] and formulate a perceived video model used in the optimization. Compared to this method in our approach we work with high-resolution videos as input to best possibly keep the originally available details.…”

“…Visual Attention In [5] and [6] optical flow is used for optimizing ADRE. However, movement is not the only attractor for visual attention.…”

“…A specialized user interface allows to interactively change the optimization parameters within the video for artistic guidance of the optimization. In contrast to [5] and [6], our approach enables apparent resolution enhancement even for scenes that do not contain any movement and for which typical optical flow computations are difficult or impossible. We encourage the reader to check the accompanying video and demo application to get a better impression of the resolution enhancement effect.…”

63‐3: Real‐time Spatial‐based Projector Resolution Enhancement

“…A typical model consists of decomposing the high-resolution image to multiple low-resolution sub-images that are projected with an offset at a higher frame rate. [1,2,3]. The sub-images are then merged by the human visual system into a single high-resolution image that approximates the original high-resolution source.…”