How a Hyper-actor directs Avatars in Virtual Shadow Theater

Abstract

After an overview of the use of digital shadows in computing science research projects with cultural and social impacts and a focus on recent researches and insights on virtual theaters, this paper introduces a research mixing the manipulation of shadow avatars and the building of a virtual theater setup inspired by traditional shadow theater (or "castelet" in french) in a mixed reality environment. It describes the virtual 3D setup, the nature of the shadow avatars and the issues of directing believable interactions between virtual avatars and physical performers on stage. Two modalities of shadow avatars direction are exposed. Some results of the research are illustrated in two use cases: the development of theatrical creativity in mixed reality through pedagogical workshops; and an artistic achievement in The Shadow performance, after H. C. Andersen.

AKN_Regie: bridging digital and performing arts

“…These developments have led to the creation of ready-to-use avatar libraries with AKN_Regie, and a method for equipping any avatar importable into UE and making it controllable with AKN_Regie. The third version of AKN_Regie incorporated the development of a specific animation player, the Salient-Idle Player, which enables fine control of avatars without using real-time motion capture, but by arranging pre-recorded animations with an approach combining theater and video games [16]. It is worth noting that developing these functionalities in AR3_Regie parent blueprint requires a confirmed level of programming with Blueprint visual language.…”

“…In summer of 2018 a second version of the plugin was released with a detailed documentation of the programming architecture on one side, and user step-by-step tutorials on the other side. Available on [16] the documentation is addressed to the theatrical students and theatre-makers who want to integrate the system in their work and for those who are willing to improve it within Blueprint environment. The documentation and the second version were successfully tested between the 2018 and 2019 within the staging of Patrick and Venus, written and directed by Anastasiia Ternova where she invited two theatre-makers without programming skills to take a total responsibility for the digital part of the performance and they achieved a satisfying autonomy using the motion capture device and AKN_Regie [39] (figure 11).…”

“…The performance is played by one actor and 5 avatars animated with pre-recorded movements controlled by the Salient-Idle player. After having performed a salient movement, each avatar is programmed to pass to a smooth idle action that it maintains for the necessary time until the next cue is triggered [16]. This player is also used for the creation of The Wizard without Shadow, a Scottish fairy tale, adapted and directed by Anastasiia Ternova in 2021 [38] (figure 13) and Les Solitudes de Donald Crowhurst, written and directed by Ce cile Roque-Alsina [39].…”

The Shadow: Coevolution Processes Between a Director, Actors and Avatars

“…Figure 5 shows part of the blueprint programming that alternates Salient and Idle animations. [19] provides a more detailed description of this programming. Implemented under the name of Salient-Idle Player (SIP), the functionality enabled the construction of 50 minutes of continuous stage play for 5 characters, creating the illusion of their autonomy from the audience's point of view.…”

“…The shadow avatar has no autonomy, it is inhabited by the mocaptor in the first presence effect circle (arrow 3 in Figure 10). All the animations are then processed with SIP (see section 4) to form a score that the narrator triggers cue by cue during the performance [19].…”