Masked-piper: Masking personal identities in visual recordings while preserving multimodal information

Owoyele, Babajide; Trujillo, James P.; Melo, Gerard de; Pouw, Wim

doi:10.31234/osf.io/bpt26

Cited by 3 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All signals were sampled at, or upsampled to, 2,500 Hz. For the video data, we used Mediapipe [20] to track the skeleton and facial movements, which is implemented in Masked-piper which we also use for masking the videos [21]. The videobased motion tracking is used to determine the the peak speed of movement, and then locate other signal peaks around them (and to centralize timeseries relative to the kinematic landmarks, such as peak speed, or movement onset).…”

Section: Methodsmentioning

confidence: 99%

Arm movements increase acoustic markers of expiratory flow

Werner,

Selen,

Pouw

2024

Preprint

Self Cite

View full text Add to dashboard Cite

The gesture-speech physics theory suggests that there are biomechanical interactions of the voice with the whole body, driving speech to align fluctuations in loudness and F0 with upper-limb movement. This exploratory study offers a possible falsification of the gesture-speech physics theory, which would predict effects of upper-limb movement on voice as well as respiration. We therefore investigate co-movement expiration. Seventeen participants were asked to produce a continuous exhalation for several seconds. After 3s, they execute one of five within-subject movement conditions with their arm with and without a wrist weight (no movement, elbow flexion, elbow extension, internal arm rotation, external arm rotation). We analyzed the smoothed amplitude envelope of the acoustic signal in relation to arm movement. Compared to no movement, all four movements lead to higher positive peaks in the amplitude peaks, while weight did not influence the amplitude. We also found that across movement conditions, positive amplitude peaks are structurally timed relative to peaks in kine-matics (speed, acceleration). We conclude that the reason why upper-limb movements affect voice loudness is still best understood through gesture-speech physics theory, where upper-limb movements affect the voice directly by modulating sub-glottal pressures. Multimodal prosody is therefore partly literally embodied.

show abstract

Section: Methodsmentioning

confidence: 99%

Arm movements increase acoustic markers of expiratory flow

Werner,

Selen,

Pouw

2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The participants are recorded via a videocamera (Logitech StreamCam), sampling at 60 frames per second. We used Mediapipe 9 , to track the skeleton and facial movements, which is implemented in Masked-piper which we also use for masking the videos 10 .…”

Section: Video and Kinematicsmentioning

confidence: 99%

Postural and muscular effects of upper-limb movements on voicing

Pouw

Burchardt

Selen

2023

Preprint

View full text Add to dashboard Cite

Voice production can be a whole-body affair: Upper limb movements physically impact the voice in steady-state vocalization, speaking, and singing. This is supposedly due to biomechanical impulses on the chest-wall, affecting subglottal pressure. Unveiling such biomechanics is important, as humans gesture with their hands in a synchronized way with speaking. Here we assess biomechanical interactions between arm movements and the voice, by measurement of key (respiratory-related) muscles with electromyography (EMG) during different types of upper limb movement while measuring the bodys center of mass. We show that gesture-related muscle activations scale with positive peaks in the voices amplitude. Some of these muscles also strongly associate with changes in the center mass, confirming that gesture-vocal coupling partly arises due to posture-related muscle activity. If replicated, these results suggest an evolutionary ancient gesture-vocal connection at the level of biomechanics. These preliminary results will support a pre-registration of analyses for a larger-scale confirmatory study.

show abstract

“…The participants are recorded via a video camera (Logitech StreamCam), sampling at 60 frames per second. We used Mediapipe (32) to track the skeleton and facial movements, which is implemented in Masked-piper which we also use for masking the videos (33). The motion-tracked skeleton, specifically the wrist of the dominant hand, is used to estimate movement initiation, peak speed, and the end of the movement.…”

Section: Supporting Information1mentioning

confidence: 99%

The human voice aligns with whole-body kinetics

Pouw,

Werner,

Burchardt

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Humans use their voice concurrently with upper limb movements, known as hand gestures. Recently it has been shown that fluctuations in intensity and the tone of the human voice synchronizes with upper limb movement (including gesticulation). In this research direct evidence is provided that the voice changes with arm movements because it interacts with whole-body muscle activity (measured through surface EMG and postural measurements). We show that certain muscles (e.g., pectoralis major) that are associated with posture and upper limb movement are especially likely to interact with the voice. Adding wrist weights to increase the mass of the moving upper limb segment led to increased coupling between movement and voice. These results show that the voice co-patterns with whole-body kinetics relating to force, rather than kinematics, invoking several implications how the voice is biomechanically modeled, how it should be simulated, and importantly how the human voice must have evolved in relation to the whole-body motor system. We concluded that the human voice is animated by the kinetics of the whole body.

show abstract

Masked-piper: Masking personal identities in visual recordings while preserving multimodal information

Cited by 3 publications

References 17 publications

Arm movements increase acoustic markers of expiratory flow

Arm movements increase acoustic markers of expiratory flow

Postural and muscular effects of upper-limb movements on voicing

The human voice aligns with whole-body kinetics

Contact Info

Product

Resources

About