NRST: Non-rigid Surface Tracking from Monocular Video

Abstract

We propose an efficient method for non-rigid surface tracking from monocular RGB videos. Given a video and a template mesh, our algorithm sequentially registers the template non-rigidly to each frame. We formulate the per-frame registration as an optimization problem that includes a novel texture term specifically tailored towards tracking objects with uniform texture but fine-scale structure, such as the regular microstructural patterns of fabric. Our texture term exploits the orientation information in the micro-structures of the objects, e.g., the yarn patterns of fabrics. This enables us to accurately track uniformly colored materials that have these high frequency micro-structures, for which traditional photometric terms are usually less effective. The results demonstrate the effectiveness of our method on both general textured non-rigid objects and monochromatic fabrics.

State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

“…SfT methods commonly operate on individual images, and although they provide 3D correspondences with a template, the reconstructions can suffer from frame‐to‐frame jitter. Deviating from this, a few works [YRCA15, HXR*18] employ an explicit temporal regularization term and ϕ‐SfT outputs temporally smooth surfaces owing to simulation. Besides, exploring joint optimization over multiple frames is promising and tractable for SfT due to advances in GPUs.…”