Teaching visual storytelling for virtual production pipelines incorporating motion capture and visual effects

Abstract

Film, television and media production are subject to consistent change due to ever-evolving technological and economic environments. Accordingly, tertiary teaching of subject areas such as cinema, animation and visual effects require frequent adjustments regarding curriculum structure and pedagogy. This paper discusses a multifaceted, cross-disciplinary approach to teaching Visual Narratives as part of a Digital Design program. Specifically, pedagogical challenges in teaching Visual Storytelling through Motion Capture and Visual Effects are addressed, and a new pedagogical framework using three different modes of moving image storytelling is applied and cited as case studies. Further, ongoing changes in film production environments and their impact on curricula for tertiary education providers are detailed, and appropriate suggestions based on tangible teaching experience are made. This paper also discusses the advantages of teaching Motion Capture in the context of immersive environments.Keywords: motion capture, visual storytelling, narrative, visual effects, curriculum Outline and BackgroundVisual Storytelling has been used in various contexts, from marketing [Mileski et al. 2015] to film [McClean 2007] and even games [Howland et al. 2007]. While the Digital Design program at Auckland University of Technology utilizes Visual Narratives in different contexts throughout the curriculum, the focus of this paper is on film and immersive environments.Teaching Visual Storytelling or Visual Narratives poses significant challenges to educators and students alike. The topic requires a * e-mail:gbennett@aut.ac.nz † e-mail:jkruse@aut.ac.nz solid theoretical foundation, and could traditionally only be explored through theory and examples in a lecture/lab style context. Particularly programs that aim to deliver content in a studio-based environment suffer from the complexity and cost-time-constraints inherently part of practical inquiry into storytelling through short film production or visual effects animation. Further, due to the structure and length of Film, Visual Effects and Digital Design degrees, there is normally only time for a single facet of visual narrative to be addressed, for example a practical camera shoot, or alternatively a visual effects or animation project. This means that comparative exploratory learning is usually out of the question, and students might only take a singular view on technical and creative story development throughout their undergraduate years. Normally only postgraduate study offers sufficient time to understand and explore multiple facets of filmic narratives. Film production pipelines in general, and visual effects workflows in particular, have seen a significant paradigm-shift in the past decade. Traditional workflows are based on frequent conversions between two dimensional images and three dimensional data sets [Okun and Zwerman 2010]. For example, the acquisition of camera images (digital or on film) projects a 3D world through a lens on to a 2D camera plane (sensor or f...

A survey on the pipeline evolution of facial capture and tracking for digital humans

“…Raytracing has improved hair simulation, subsurface scattering shaders, and other benefits for the render of digital humans. Performing retargeting before streaming into a rendering engine is also possible through standardized connections, like the ones in Motion Builder by Autodesk, that connect across several real-time technologies from companies such as Vicon, Optitrack, or Xsens [14]. Some options like Shogun Live, and Unreal Engine 5 IK retargeter [54], allow skipping Motion Builder by directly connecting to Unreal Engine.…”

Active Learning and Teaching through Digital Technology and Live Performance: ‘Choreographic Thinking’ as Art Practice in the Tertiary Sector

“…In 2013, AUT opened the Sir Paul Reeves building, an extension to the city campus featuring a MoCap studio and a performance studio. The MoCap studio is a vital facility for the teaching of undergraduate and postgraduate students in Digital Design degrees at AUT (Bennett & Kruse ), and for generating revenue from industry projects (Auckland University of Technology ). The room is equipped with 24 Motion Analysis cameras that can track up to 6 ‘actors’ in full body marker sets in a 4x4x2 metre volume (Motion Analysis Corporation ).…”

Linguagem cinematográfica na formação docente: o ético e a estética na educação

“…Desse modo, a experiência estética com a linguagem cinematográfica, na formação dos professores aproxima-os ao sensível, ao som as imagens e à materialidade do cinema constituindo esquemas de percepções, gostos, julgamentos e imaginações, bem como fomenta uma leitura criativa, crítica e sensível do mundo (Wheeler, 2018;Genovard Roselló, Casulleras Femenia, Gotzens Busquets, 2010;Çengelci, 2013;Areiza Pérez e Betancur Valencia, 2011;Fischer, 2009;Clarembeaux, 2010;Bennett e Kruse, 2015;González Blasco, Moreto e Pessini, 2018;Costa, Branquinho e Pereira 2018).…”