Phase-based video motion processing

Abstract

a) Source (b) Linear [Wu et al. 2012] (c) Phase-based (this paper) x t Figure 1: Motion magnification of a crane imperceptibly swaying in the wind. (a) Top: a zoom-in onto a patch in the original sequence (crane) shown on the left. Bottom: a spatiotemporal XT slice of the video along the profile marked on the zoomed-in patch. (b-c) Linear [Wu et al. 2012] and phase-based motion magnification results, respectively, shown for the corresponding patch and spatiotemporal slice as in (a). The previous, linear method visualizes the crane's motion, but amplifies both signal and noise and introduces artifacts for higher spatial frequencies and larger motions, shown by the clipped intensities (bright pixels) in (b). In comparison, our new phase-based method supports larger magnification factors with significantly fewer artifacts and less noise (c). The full sequences are available in the supplemental video. AbstractWe introduce a technique to manipulate small movements in videos based on an analysis of motion in complex-valued image pyramids. Phase variations of the coefficients of a complex-valued steerable pyramid over time correspond to motion, and can be temporally processed and amplified to reveal imperceptible motions, or attenuated to remove distracting changes. This processing does not involve the computation of optical flow, and in comparison to the previous Eulerian Video Magnification method it supports larger amplification factors and is significantly less sensitive to noise. These improved capabilities broaden the set of applications for motion processing in videos. We demonstrate the advantages of this approach on synthetic and natural video sequences, and explore applications in scientific analysis, visualization and video enhancement.

Riesz pyramids for fast phase-based video magnification

“…These motions require computational amplification to be revealed [10,12,19,21]. Manipulating the local phase in coefficients of a complex steerable pyramid decomposition of an image sequence is an effective, robust method of amplifying small motions in video [19], but complex steerable pyramids are very overcomplete (21 times) and costly to construct, requiring either a large number of filter taps or a frequency domain construction where care must be taken to avoid spatial wrap-around artifacts [11,15].…”

“…We present a new image pyramid representation, the Riesz pyramid, that is suitable for Eulerian phase-based video magnification, but is much less overcomplete than the complex steerable pyramid used by Wadhwa et al [16,19]. Our new representation produces motion-magnified videos of comparable quality to those produced using a complex steerable pyramid, but the videos can be processed in one quarter of the time, making it more suitable for real-time or online processing ( Figure 1).…”

“…This processing is done entirely in the spatial domain, which gives an easy way of avoiding the spatial wraparound artifacts present in the frequency domain implementation of the eight-orientation complex steerable pyramid used by Wadhwa et al [19]. Building and collapsing the Riesz pyramid is efficiently implemented because of shared computation between bands, symmetry of the filters, and because the Riesz transform is approximated by two threetap finite difference filters.…”

“…In summary, we present a new representation that can be used for video magnification that is (a) less overcom- [21] (b) and two phase-based methods, first with an eight orientation octavebandwidth complex steerable pyramid [19] (c), and second with our new Riesz pyramid (d). The quality of the video produced using our new representation (d) is comparable to that produced using the complex steerable pyramid method (c), but is approximately four times faster to compute.…”

Motion Characterization of a Dynamic Scene

“…The phase variations are processed temporally to remove or enhance minute changes over time. The processing does not involve optical flow estimation and is therefore suitable for processing of videos of scenes where optical flow based approaches may fail (Wadhwa et al, 2013).…”