Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano

Wu, Erwin; Nishioka, Hayato; Furuya, Shinichi; Koike, Hideki

doi:10.1109/wacv56688.2023.00299

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Book1

Other1

Relationship

Self Cite0

Independent2

Authors

Journals

Cited by 2 publications

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Examining the Relationship Between Playing a Chord with Expressions and Hand Movements Using MediaPipe

Oshima,

Takatsu,

Nakayama

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Examining the Relationship Between Playing a Chord with Expressions and Hand Movements Using MediaPipe

Oshima,

Takatsu,

Nakayama

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

FürElise: Capturing and Physically Synthesizing Hand Motion of Piano Performance

Wang,

Xu,

Shi

et al. 2024

SIGGRAPH Asia 2024 Conference Papers

View full text Add to dashboard Cite

Piano playing requires agile, precise, and coordinated hand control that stretches the limits of dexterity. Hand motion models with the sophistication to accurately recreate piano playing have a wide range of applications in character animation, embodied AI, biomechanics, and VR/AR. In this paper, we construct a first-of-its-kind large-scale dataset that contains approximately 10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153 pieces of classical music. To capture natural performances, we designed a markerless setup in which motions are reconstructed from multiview videos using state-of-the-art pose estimation models. The motion data is further refined via inverse kinematics using the high-resolution MIDI key-pressing data obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the collected dataset, we developed a pipeline that can synthesize physically-plausible hand motions for musical scores outside of the dataset. Our approach employs a combination of imitation learning and reinforcement learning to obtain policies for physics-based bimanual control involving the interaction between hands and piano keys. To solve the sampling efficiency problem with the large motion dataset, we use a diffusion model to generate natural reference motions, which provide high-level trajectory and fingering (finger order and placement) information. However, the generated reference motion alone does not provide sufficient accuracy for piano performance modeling. We then further augmented the data by using musical similarity to retrieve similar motions from the captured dataset to boost the precision of the RL policy. With the proposed method, our model generates natural, dexterous motions that generalize to music from outside the training dataset.

show abstract

Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano

Cited by 2 publications

References 29 publications

Examining the Relationship Between Playing a Chord with Expressions and Hand Movements Using MediaPipe

Examining the Relationship Between Playing a Chord with Expressions and Hand Movements Using MediaPipe

FürElise: Capturing and Physically Synthesizing Hand Motion of Piano Performance

Contact Info

Product

Resources

About