Expressive autonomous cinematography for interactive virtual environments

Abstract

We have created an automatic cinematography system for interactive virtual environments. This system controls a virtual camera and lights in a three-dimensional virtual world inhabited by a group of autonomous and user-controlled characters. By dynamically changing the camera and the lights, our system facilitates the interaction of human participants with this world and displays the emotional content of the digital scene.Building on the tradition of cinema, modern video games, and autonomous behavior systems, we have constructed this cinematography system with an ethologically-inspired structure of sensors, emotions, motivations, and action-selection mechanisms. Our system breaks shots into elements, such as which actors the camera should focus on or the angle it should use to watch them. Hierarchically arranged cross-exclusion groups mediate between the various options, arriving at the best shot at each moment in time. Our cinematography system uses the same approach that we use for our virtual actors. This eases the cross-over of information between them, and ultimately leads to a richer and more unified installation.As digital visualizations grow more complex, cinematography must keep pace with the new breeds of characters and scenarios. A behavior-based autonomous cinematography system is an effective tool in the creation of interesting virtual worlds. Our work takes first steps toward a future of interactive, emotional cinematography.

Automated camera planning to film robot operations

“…Geometric frame-level constraints are solved using a contraint-solving technique, in which constraints can relate to the height angle, distance to the target, and orientation of the camera. Tomlinson et al (2000) implemented an autonomous cinematography system based on the autonomous character design work of the Synthetic Characters Group at the MIT Media Lab. Their system used expressive and user-controlled characters to drive the on-the-fly selection of shots in a virtual collaborative 3D environment.…”

Automated videography for residential communications

“…While this method is interesting, it seems inappropriate for real-time video communications. There are also numerous examples in the literature [26][27][28] where cinematography is adapted to the capture of activities of virtual actors or avatars that function within virtual worlds. While the unfolding events can be unscripted and occur in real time, cinematography in virtual metaverses is not bound by real-world restrictions.…”

3D screen-space widgets for non-linear projection

“…Image-space constraints have been used to control camera animations [Blinn 1988;Gleicher and Witkin 1992], automatic camera control for teleconferencing-type applications [Drucker and Zeltzer 1995], and automatic composition [Tomlinson et al 2000;Gooch et al 2001]. Gleicher [Gleicher and Witkin 1992], Blinn [Blinn 1988], and Tomlinson [Tomlinson et al 2000] all used image-space constraints and a general purpose solver to create a camera animation.…”

“…Gleicher [Gleicher and Witkin 1992], Blinn [Blinn 1988], and Tomlinson [Tomlinson et al 2000] all used image-space constraints and a general purpose solver to create a camera animation. All of these approaches used only seven of the eleven camera parameters (position, orientation, and focal length), a small vocabulary of image constraints (point constraints, size in image, and a notion of "up") and a simple heuristic for any unconstrained parameters (keep them the same as before).…”