Selfie Video Stabilization

We present Steadiface, a new real-time face-centric video stabilization method that simultaneously removes hand shake and keeps subject's head stable. We use a CNN to estimate the face landmarks and use them to optimize a stabilized head center. We then formulate an optimization problem to find a virtual camera pose that locates the face to the stabilized head center while retains smooth rotation and translation transitions across frames. We test the proposed method on fieldtest videos and show it stabilizes both the head motion and background. It is robust to large head pose, occlusion, facial appearance variations, and different kinds of camera motions. We show our method advances the state of art in selfie video stabilization by comparing against alternative methods. The whole process runs very efficiently on a modern mobile phone (8.1 ms/frame).

show abstract

“…7. Comparison against video stabilization method in [12]. Each frame is the average of consecutive 15 frames.…”

Section: Importance Of Each Termmentioning

confidence: 99%

“…Yu and Ramamoorthi proposed an image-based method for facecentric stabilization [12]. The head motion is modeled as the 3D head center of the reconstructed head, and the background motion is tracked by dense optical flow.…”

Section: Introductionmentioning

confidence: 99%

Steadiface: Real-Time Face-Centric Stabilization On Mobile Phones

Shi

Tsai

Wang

et al. 2019

2019 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…Nevertheless, their videos are often very short and contain only one large main face, which makes them less useful for developing mobile face trackers or benchmarking object trackers in the mobile domain. Yu and Ramamoorthi [4] recently proposed a mobile dataset that contains 33 in-the-wild selfie videos. At initial look the data set would seem suitable for benchmarking trackers in mobile settings.…”

Section: A Face Datasets In the Mobile Domainmentioning

confidence: 99%

“…3) Multiple Faces (MF): It indicates whether there are more than one face in the video or not. There are usually no similar objects in one single video in object tracking datasets [12]- [14], [25] and mobile datasets [1]- [4].…”

Section: Attributes Annotationmentioning

confidence: 99%

MobiFace: A Novel Dataset for Mobile Face Tracking in the Wild

Lin

Cheng

Shen

et al. 2019

2019 14th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2019)

View full text Add to dashboard Cite

Face tracking serves as the crucial initial step in mobile applications trying to analyse target faces over time in mobile settings. However, this problem has received little attention, mainly due to the scarcity of dedicated face tracking benchmarks. In this work, we introduce MobiFace, the first dataset for single face tracking in mobile situations. It consists of 80 unedited live-streaming mobile videos captured by 70 different smartphone users in fully unconstrained environments. Over 95K bounding boxes are manually labelled. The videos are carefully selected to cover typical smartphone usage. The videos are also annotated with 14 attributes, including 6 newly proposed attributes and 8 commonly seen in object tracking. 36 state-of-the-art trackers, including facial landmark trackers, generic object trackers and trackers that we have fine-tuned or improved, are evaluated. The results suggest that mobile face tracking cannot be solved through existing approaches. In addition, we show that fine-tuning on the MobiFace training data significantly boosts the performance of deep learningbased trackers, suggesting that MobiFace captures the unique characteristics of mobile face tracking. Our goal is to offer the community a diverse dataset to enable the design and evaluation of mobile face trackers. The dataset, annotations and the evaluation server will be on https://mobiface. github.io/.

show abstract

“…With the increased demand for high-quality video using handheld devices, video stabilization has become increasingly important, as such videos always contain undesirable jitter. Many video stabilization methods have been proposed to eliminate jitter in unstable videos for a better visual experience [31,5,38,36,37], and can facilitate many other computer vision tasks [3,27,42,4,41,40].…”

Section: Introductionmentioning

confidence: 99%

Out-of-boundary View Synthesis Towards Full-Frame Video Stabilization

Xu¹,

Zhang²,

Tao³

2021

Preprint

View full text Add to dashboard Cite

Warping-based video stabilizers smooth camera trajectory by constraining each pixel's displacement and warp stabilized frames from unstable ones accordingly. However, since the view outside the boundary is not available during warping, the resulting holes around the boundary of the stabilized frame must be discarded (i.e., cropping) to maintain visual consistency, and thus does leads to a tradeoff between stability and cropping ratio. In this paper, we make a first attempt to address this issue by proposing a new Outof-boundary View Synthesis (OVS) method. By the nature of spatial coherence between adjacent frames and within each frame, OVS extrapolates the out-of-boundary view by aligning adjacent frames to each reference one. Technically, it first calculates the optical flow and propagates it to the outer boundary region according to the affinity, and then warps pixels accordingly. OVS can be integrated into existing warping-based stabilizers as a plug-and-play module to significantly improve the cropping ratio of the stabilized results. In addition, stability is improved because the jitter amplification effect caused by cropping and resizing is reduced. Experimental results on the NUS benchmark show that OVS can improve the performance of five representative state-of-the-art methods in terms of objective metrics and subjective visual quality. 1

show abstract

Selfie Video Stabilization

Cited by 38 publications

References 21 publications

Steadiface: Real-Time Face-Centric Stabilization On Mobile Phones

Steadiface: Real-Time Face-Centric Stabilization On Mobile Phones

MobiFace: A Novel Dataset for Mobile Face Tracking in the Wild

Out-of-boundary View Synthesis Towards Full-Frame Video Stabilization

Contact Info

Product

Resources

About