Flexible flow for 3D nonrigid tracking and shape recovery

Abstract

We introduce linear methods for model-based tracking of nonrigid 3D objects and for acquiring such models from video. 3D motions and flexions are calculated directly from image intensities without information-lossy intermediate results. Measurement uncertainty is quantified and fully propagated through the inverse model to yield posterior mean (PM) and mode (MAP) pose estimates. A Bayesian framework manages uncertainty, accommodates priors, and gives confidence measures. We obtain highly accurate and robust closed-form estimators by minimizing information loss from non-reversible (inner-product and least-squares) operations, and, when unavoidable, performing such operations with the appropriate error norm. For model acquisition, we show how to refine a crude or generic model to fit the video subject. We demonstrate with tracking, model refinement, and super-resolution texture lifting from low-quality low-resolution video.This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Information Technology Center America; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Information Technology Center America. All rights reserved.

Perspective Nonrigid Shape and Motion Recovery

“…The rank constraint implied by (4) has been the basis for existing projective NRSM algorithms. As shown in [12], when the depths are known, the shape coefficients and shape basis may be computed from the factorization of W using a factorization technique similar to that in [8] for affine cameras.…”

“…As suggested in [3][4][5][6][7], we assume that the P points deform as a linear combination of a fixed set of K rigid shape bases with time varying coefficients. That is,…”

“…This has motivated the development of a family of methods where a moving affine calibrated camera observes a nonrigid shape that deforms as a linear combination of K rigid shapes with time varying coefficients [3][4][5][6][7][8]. This assumption allows one to recover nonrigid shape and motion (NRSM) using extensions of the classical rigid factorization algorithm of Tomasi and Kanade [9].…”

Automatic Facial Expression Recognition

“…Depending on the approach the morphing may use planar models of the face [34,29], cylindrical models of the face [27], ellipsoid models [38], 2D active appearance models based on a triangulated mesh [32], or 3D deformable models [34,5,55].…”

“…Other systems use a larger number of less reliable but faster feature detectors datasets. Finally, some systems track a large number of very simple features that are trained on a specific person [5,55,34].…”