TimToShape: Supporting Practice of Musical Instruments by Visualizing Timbre with 2D Shapes based on Crossmodal Correspondences

Arai, Kota; Hirao, Yutaro; Narumi, Takuji; Nakamura, Tomohiko; Takamichi, Shinnosuke; Yoshida, Shigeo

doi:10.1145/3581641.3584053

Search citation statements

Order By: Relevance

Paper Sections

Select...

Sound Symbolism1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Preprint1

Article1

Relationship

Self Cite0

Independent2

Authors

Journals

Cited by 2 publications

(1 citation statement)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the recent studies have extended sound symbolization through timbre analysis. In [25], music timbre was mapped onto two-dimensional spaces, which is then used to guide the generation of shapes that reflect users' preferences.…”

Section: Sound Symbolismmentioning

confidence: 99%

Visual Signatures for Music Mood and Timbre

Wang,

Sourin

2024

Preprint

View full text Add to dashboard Cite

Existing research on music visualization has primarily focused on creating animated visual illustrations to accompany the music being played based on fundamental attributes such as sound frequency or music structure, whereas the higher-level features, including mood and timbre, are mostly overlooked. In this paper, we propose visual signatures to describe the higher-level attributes of music, where the content and the color palette of the visual signatures are controlled by the music mood and timbre, respectively. We expect that the users with different cultural and educational backgrounds will be able to easily interpret the meaning of sound with the proposed visual signatures. In our work, we used a contrastive learning neural network for mood classification and an audio Transformer for timbre classification. The performance of the music classification models is examined by their accuracy, while multiple generated images are displayed to showcase the feasibility of visual signatures.

show abstract

Section: Sound Symbolismmentioning

confidence: 99%