Interactive Body-Driven Graphics for Augmented Video Performance

Abstract

Figure 1: Our system is a presentation tool that lets the performer interactively control the graphical elements using a wide range of hand gestures and body postures for a real-time storytelling experience. (a) An umbrella is attached to the presenter's hand, (b) controlling a custom-made slider to change the humidity chart, (c) directly manipulating a virtual sketch (cross arrow) to change the timeline of a data chart, and (d) pointing and flicking gestures (blue arrows) to move the temperature arrow, which manipulates the sun's opacity.

PoseTween

“…For example, a subject's perfor mance for manipulating physical puppets can be captured to transform corresponding 3D virtual puppets for making 3D animations [13]. Recent gesture-based frameworks [3,26] first map a number of hand gestures or body gestures to the parameters of virtual objects (e.g. opacity, motion speed, etc.…”

“…PoseTween differs from existing tracking-based systems (e.g., Octi and [26]) in that it allows a more flexible and controllable movement of virtual objects. This enables more complex interactions between an object and a subject, and increases the expressiveness of body-driven visual effects (see a comparison in Figure 8).…”

“…Several existing works introduce human factors in making animations. One group of methods allows users to map a few hand gestures [18] or body gestures [3,26,28] to the parameters of target animated objects in an authoring process, and the parameters can then be changed in real time by the authored gestures in a perfor mance process. However, such gesture-based methods require a gesture recognition model, which prevents them from incor porating new user-customized gestures easily.…”

“…Since a user can only perform a limited number of gestures at one time, the parameters that can be changed simultaneously are thus also limited. In addition, even if the gestures are well-categorized in terms of expressiveness [26], they do not necessarily have semantic correlations with the parameters of objects to be animated and thus users need to memorize the customized mappings for live performance.…”

“…The association inference that finds the driving body parts of an object is conducted automatically when users edit the keyframes of an object (details in Section "Inference of Associations"). An alternative approach is to let users di rectly indicate object-to-body associations, as in [26]. Such direct graphics-to-body associations might work well when an object is associated to a fixed body part during animation.…”