Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles

Joubert, N.; E, Jane L.; Goldman, Dan B; Berthouzoz, Floraine; Roberts, Michael J.; Landay, James A.; Hanrahan, Pat

doi:10.48550/arxiv.1610.01691

Cited by 17 publications

(28 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Related Workmentioning

confidence: 99%

“…Autonomous Aerial Filming: Recent research works [21,22,8,9,15,16,13,11] enable flexible human-drone interactions in an aerial filming task. For instance, in [21,22,8,9,15,13,16], the users are allowed to specify the subject size, viewing angle and position on the screen to generate quadrotor's motion automatically. However, these techniques essentially move a camera to a target pose based on the demand specified by users; therefore, the aesthetic quality of the video highly relies on the user's input.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

One-Shot Imitation Drone Filming of Human Motion Videos

Huang

Dang

Chen

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Imitation learning has been applied to mimic the operation of a human cameraman in several autonomous cinematography systems [4,3,5,12]. To imitate different filming styles, existing methods train multiple models, where each model handles a particular style and requires a significant number of training samples. As a result, existing methods can hardly generalize to unseen styles. In this paper, we propose a framework, which can imitate a filming style by "seeing" only a single demonstration video of the same style, i.e., one-shot imitation filming. This is done by two key enabling techniques: 1) feature extraction of the filming style from the demo video, and 2) filming style transfer from the demo video to the new situation. We implement the approach with deep neural network and deploy it to a 6 degrees of freedom (DOF) real drone cinematography system by first predicting the future camera motions, and then converting them to the drone's control commands via an odometer. Our experimental results on extensive datasets and showcases exhibit significant improvements in our approach over conventional baselines and our approach can successfully mimic the footage with a unseen style.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

One-Shot Imitation Drone Filming of Human Motion Videos

Huang

Dang

Chen

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…This enables the drone to move to a position where requirements from canonical static shots, like the rule of thirds, or other composition guidelines are satisfied [7]. Besides the aesthetics, it is often essential to transition between positions in a secure way (keeping a safe distance from the targets) [8].…”

Section: Related Workmentioning

confidence: 99%

CineMPC: Controlling Camera Intrinsics and Extrinsics for Autonomous Cinematography

Pueyo¹,

Montijano²,

Murillo³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present CineMPC, an algorithm to autonomously control a UAV-borne video camera in a nonlinear MPC loop. CineMPC controls both the position and orientation of the camera-the camera extrinsics-as well as the lens focal length, focal distance, and aperture-the camera intrinsics. While some existing solutions autonomously control the position and orientation of the camera, no existing solutions also control the intrinsic parameters, which are essential tools for rich cinematographic expression. The intrinsic parameters control the parts of the scene that are focused or blurred, and the viewers' perception of depth in the scene. Cinematographers commonly use the camera intrinsics to direct the viewers' attention through the use of focus, to convey suspense through telephoto views, inspire awe through wide-angle views, and generally to convey an emotionally rich viewing experience. Our algorithm can use any existing approach to detect the subjects in the scene, and tracks those subjects throughout a user-specified desired camera trajectory that includes camera intrinsics. CineMPC closes the loop from camera images to UAV trajectory in order to follow the desired relative trajectory as the subjects move through the scene. The cinematographer can use CineMPC to autonomously record scenes using the full array of cinematographic tools for artistic expression.

show abstract

“…These key frames can be predefined by a user [23], [24] or sampled intelligently from a set of template shot types [2], [22]. [25] automatically records videos of two subjects by matching the view with predefined templates for scenes with two actors. Instead of defining key frames, imitation learning [26] can be used to learn camera trajectories directly from films.…”

Section: Introductionmentioning

confidence: 99%

“…This challenge limits the ease of integrating learned aesthetics models into drone cinematography. As such, a common property of the many autonomous cinematography works is that the captured video focuses on tracking human subjects [1], [2], [4], [11], [22], [25]. Commercial drones (such as Skydio) also focus on autonomous cinematography for a single user by filming them during activities like skiing or mountain biking.…”

Section: Introductionmentioning

confidence: 99%

AutoPhoto: Aesthetic Photo Capture using Reinforcement Learning

Hadi¹,

Lin²,

Bala³

2021

Preprint

View full text Add to dashboard Cite

The process of capturing a well-composed photo is difficult and it takes years of experience to master. We propose a novel pipeline for an autonomous agent to automatically capture an aesthetic photograph by navigating within a local region in a scene. Instead of classical optimization over heuristics such as the rule-of-thirds, we adopt a data-driven aesthetics estimator to assess photo quality. A reinforcement learning framework is used to optimize the model with respect to the learned aesthetics metric. We train our model in simulation with indoor scenes, and we demonstrate that our system can capture aesthetic photos in both simulation and real world environments on a ground robot. To our knowledge, this is the first system that can automatically explore an environment to capture an aesthetic photo with respect to a learned aesthetic estimator. Source code is at https://github.com/HadiZayer/AutoPhoto

show abstract

Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles

Cited by 17 publications

References 20 publications

One-Shot Imitation Drone Filming of Human Motion Videos

One-Shot Imitation Drone Filming of Human Motion Videos

CineMPC: Controlling Camera Intrinsics and Extrinsics for Autonomous Cinematography

AutoPhoto: Aesthetic Photo Capture using Reinforcement Learning

Contact Info

Product

Resources

About