An interactive staging-and-shooting solver for virtual cinematography

Abstract

Research in virtual cinematography often narrows the problem down to computing the optimal viewpoint for the camera to properly convey a scene's content. In contrast we propose to address simultaneously the questions of placing cameras, lights, objects and actors in a virtual environment through a high-level specification. We build on a staging language and propose to extend it by defining complex temporal relationships between these entities. We solve such specifications by designing pruning operators which iteratively reduce the range of possible degrees of freedom for entities while satisfying temporal constraints. Our solver first decomposes the problem by analyzing the graph of relationships between entities and then solves an ordered sequence of sub-problems. Users have the possibility to manipulate the current result for fine-tuning purposes or to creatively explore ranges of solutions while maintaining the relationships. As a result, the proposed system is the first staging-and-shooting cinematography system which enables the specification and solving of spatio-temporal cinematic layouts.

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

“…Among the most relevant ones, Ye et al [57] focus on the language mapping between the action descriptions and avatar animation instead of the visual content and PIzzi et al [37] only generate sketchstyle static images. Besides, intelligent creation tools are in great demand as they can help users efficiently create customized dynamic content, e.g., video, animation [27,28]. Some researchers focus on several key steps, such as frame composition [60], shot selection [23,25], shot cut suggestion [35].…”