Spatio-Temporal Human Shape Completion With Implicit Function Networks

Abstract

We address the problem of inferring a human shape from partial observations, such as depth images, in temporal sequences. Deep Neural Networks (DNN) have been shown successful to estimate detailed shapes on a frame-by-frame basis but consider yet little or no temporal information over frame sequences for detailed shape estimation. Recently, networks that implicitly encode shape occupancy using MLP layers have shown very promising results for such single-frame shape inference, with the advantage of reducing the dimensionality of the problem and providing continuously encoded results. In this work we propose to generalize implicit encoding to spatio-temporal shape inference with spatio-temporal implicit function networks or STIF-Nets, where temporal redundancy and continuity is expected to improve the shape and motion quality. To validate these added benefits, we collect and train with motion data from CAPE for dressed humans, and DFAUST for body shapes with no clothing. We show our model's ability to estimate shapes for a set of input frames, and interpolate between them. Our results show that our method outperforms existing state of the art methods, in particular the singleframe methods for detailed shape estimation.

Pyramidal Signed Distance Learning for Spatio-Temporal Human Shape Completion

“…The representation of the predicted surface can be explicit [19,40,52] or implicit which allows for changes in the topology [11,35]. This approach has been used for non-articulated objects [24,30,47,33,37,13] and articulated human body [42,43,12,20,16,51,40]. The implicit surface can be represented using a (truncated) signed distance function, an unsigned distance function [13] or an occupancy function as in IF-Net [12].…”

“…This strategy enforces by construction temporal consistency, but it also prevents temporal information to benefit to the shape model. STIF [51] addresses the problem of temporal integration using a recurrent GRU [14,15] layer to aggregate information. Such layer can however only be used at the low-resolution features and hence surface details do not propagate from one frame to the next.…”

“…[48,44]) is to build a feature hierarchy or feature pyramid [23] to extract various feature scale levels. The recent literature [51] shows that a measurable improvement to this strategy, in the context of temporal input images sequences, is to link the coarsest layer in each input image's pyramid with a bidirectional recurrent GRU component [14,15], instead of building independent per-frame pyramids. This allows the model to learn common global characteristics and their mutual updates at a modest computational cost.…”

“…On the other hand, neural network based approaches, e.g. NDF [13], IF-Net [12] or STIF [51], use implicit representations to model the observed surfaces. Such representations give access to a larger set of shapes, although this set is still bounded by the coverage in the training data.…”

“…This strategy applies to the spatial domain as well as to the temporal domain. We investigate the temporal dimension since redundancy over time can contribute to shape completion, as demonstrated in [51]. This dimension naturally integrates into the pyramidal spatio-temporal model we present.…”