Landscapes of coarticulation: The co-structuring of gesture-vocal dynamics in Karnatak music performance

Abstract

In vocal performance contexts, vocalists tend to gesture in ways that show both similarities and idiosyncrasies. We present a quantitative analyses and visualisation pipeline that characterises the multidimensional codependencies of spontaneous body movements and vocalisations in vocal performers. We apply this pipeline to a dataset of performances within the Karnatak music tradition of South India, including audio and motion tracking data, which we openly publish with this report. Our results show that the time-varying features of head and hand gestures tend to be more similar when the concurrent vocal time-varying features are also more similar. While for each performer we find clear co-structuring of sound and movement, they each show their own characteristic salient dimensions (e.g., hand position, head acceleration) on which movement is coarticulated with singing. Our analyses thereby provide a computational characterisation of the unique multimodal coarticulations with singing for each performer. The results support our conceptual contribution of widening the conception of coarticulation, from within a ‘modality’ (e.g., speech articulator positions, joint angles in reaching), to a multimodal coarticulation constrained by both physiological and aesthetic ‘control parameters’ that reduce degrees of freedom of the multimodal performance such that motifs that sound alike tend to co-structure with gestures that move alike.