Camera keyframing with style and control

Jiang, Huiyun; Christie, Marc; Xi, Wang; Liu, Libin; Wang, Bin; Chen, Baoquan

doi:10.1145/3478513.3480533

Cited by 16 publications

(13 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such a system encodes an example sequence to a latent style code with a gating network and generates a camera sequence through generative experts that account for both the encoded style code and keyframe constraints. To perform our comparison (see Tab 1), we first took ground truth camera sequences as inputs to the SoTA system (LSTM+Gating [JCW*21]), to serve as a reference. Then for the other models, we used the textual descriptions linked to the ground truth sequences.…”

Section: Results and Experimentsmentioning

confidence: 99%

Cinematographic Camera Diffusion Model

Jiang,

Wang,

Christie

et al. 2024

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

Designing effective camera trajectories in virtual 3D environments is a challenging task even for experienced animators. Despite an elaborate film grammar, forged through years of experience, that enables the specification of camera motions through cinematographic properties (framing, shots sizes, angles, motions), there are endless possibilities in deciding how to place and move cameras with characters. Dealing with these possibilities is part of the complexity of the problem. While numerous techniques have been proposed in the literature (optimization‐based solving, encoding of empirical rules, learning from real examples,…), the results either lack variety or ease of control.In this paper, we propose a cinematographic camera diffusion model using a transformer‐based architecture to handle temporality and exploit the stochasticity of diffusion models to generate diverse and qualitative trajectories conditioned by high‐level textual descriptions. We extend the work by integrating keyframing constraints and the ability to blend naturally between motions using latent interpolation, in a way to augment the degree of control of the designers. We demonstrate the strengths of this text‐to‐camera motion approach through qualitative and quantitative experiments and gather feedback from professional artists. The code and data are available at https://github.com/jianghd1996/Camera-control.

show abstract

Section: Results and Experimentsmentioning

confidence: 99%

Cinematographic Camera Diffusion Model

Jiang,

Wang,

Christie

et al. 2024

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

show abstract

“…Later works tend to directly imitate exemplar videos to generate similar camera trajectories with SfM [42]. Jiang et al [22] use deep neural networks to extract camera behaviors from real movie clips based on toric space [26], which is helpful in imitating camera motion [58] and keyframing [21]. However, the inaccuracy of SfM and toric space estimation may severely affect their performance.…”

Section: Related Workmentioning

confidence: 99%

“…Still, two significant problems need to be further addressed in order to fulfill this task with enough high quality. First, most existing works [6,21,22,44] only study the camera behavior under fixed plots and environments that require heavy manual force to modify. Secondly, the full action space of camera movements is too large to allow effective per-frame decisions [49].…”

Section: Introductionmentioning

confidence: 99%

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

Rao¹,

Jiang²,

Guo³

et al. 2023

Preprint

View full text Add to dashboard Cite

Dynamic Storyboard 2 Dynamic Storyboard 1 Virtual Environment 📷 Camera Scripts 2: Dolly Full High-angle 📷 Camera scripts 1: Push Medium Eye-level 📖 Story scripts: Jane and Jack are arguing in living room Figure 1. We present Virtual Dynamic Storyboard (VDS) that takes user input story and camera scripts and automatically composes dynamic storyboards in an engine-based virtual environment for pre-visualization. Here we show two results produced by VDS with top-ranked scores of quality. Video demos can be found in the supplementary.

show abstract

“…The same authors extend the work in [9] with an algorithm that classifies and recreates the style that defines the relative pose between camera and subject from an input sequence, using multiple layers of imitation learning. The work in [10] presents a framework that uses machine learning to move a drone to reproduce the style of a reference movie clip, passing through a set of given camera viewpoints. The paper [11] considers an imitation learning framework where a network predicts camera motions using a dataset of human-based videos recorded by professional cinematographers.…”

Section: Related Workmentioning

confidence: 99%