Multimodal Music Information Processing and Retrieval: Survey and Future Challenges

Simonetta, Federico; Ntalampiras, Stavros; Avanzini, Federico

doi:10.1109/mmrp.2019.00012

Cited by 50 publications

(47 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worth underlining that the field of multimodal analysis, that in recent years has raised more and more attention by the music research community, can benefit from the adoption of the IEEE 1599 standard is. According to [19], which extensively reviewed the Multimodal Music Information Retrieval (MIR), the concept of modality can be defined as a music representation digitized in a particular place and time; multiple modalities can originate by digitizing music information in different places or times. This renewed interest in multimodal analysis was due to the occurrence of multiple factors, such as an increased computational power allowing for more complex approaches in everyday computers and the spread of machine-learning methods which are able to deal with heterogeneous kinds of data.…”

Section: Discussion and Final Remarksmentioning

confidence: 99%

On the Adoption of Standard Encoding Formats to Ensure Interoperability of Music Digital Archives: The IEEE 1599 Format

Ludovico

Baratè

Simonetta

et al. 2019

6th International Conference on Digital Libraries for Musicology

Self Cite

View full text Add to dashboard Cite

With this paper, we want to stimulate the discussion about technologies for inter-operation between various music datasets and collections. Among the many standards for music representation, IEEE 1599 is the only one which was born with the exact purpose of representing the heterogeneous structures of music documents, granting full synchronization of all the different aspects of music (audio recordings, sheet music images, symbolic representations, musicological analysis, etc). We propose the adoption of IEEE 1599 as an interoperability framework between different collections for advanced music experience, musicological applications, and Music Information Retrieval (MIR). In the years to come, the format will undergo a review process aimed at providing an updated/improved version. It is now the perfect time, for all the stakeholders, to come together and discuss how the format can evolve to better support their requirements, enhancing its descriptive strength and available tools. Moreover, this standard can be profitably applied to any field that requires multi-layer and synchronized descriptions. CCS CONCEPTS • Applied computing → Sound and music computing; • Information systems → Digital libraries and archives; • General and reference → Computing standards, RFCs and guidelines.

show abstract

Section: Discussion and Final Remarksmentioning

confidence: 99%

On the Adoption of Standard Encoding Formats to Ensure Interoperability of Music Digital Archives: The IEEE 1599 Format

Ludovico

Baratè

Simonetta

et al. 2019

6th International Conference on Digital Libraries for Musicology

Self Cite

View full text Add to dashboard Cite

show abstract

“…These techniques are central for the development of machine learning models able to process and relate data from multiple modalities, and thereby gain an in-depth understanding of complex phenomena that humans experience multimodally (Baltrusaitis et al, 2019). Particularly, such techniques are said to have considerable advantages over unimodal ones for the analysis of music, as several music processing tasks -including similarity computation, classification in high-level categories describing emotion or expressivity, structural segmentation, and others -can benefit profoundly from multimodal approaches (Simonetta et al, 2019).…”

Section: Multimodal Music Representation and Analysismentioning

confidence: 99%

Method Development for Multimodal Data Corpus Analysis of Expressive Instrumental Music Performance

Visi

Östersjö

et al. 2020

Front. Psychol.

View full text Add to dashboard Cite

Musical performance is a multimodal experience, for performers and listeners alike. This paper reports on a pilot study which constitutes the first step toward a comprehensive approach to the experience of music as performed. We aim at bridging the gap between qualitative and quantitative approaches, by combining methods for data collection. The purpose is to build a data corpus containing multimodal measures linked to high-level subjective observations. This will allow for a systematic inclusion of the knowledge of music professionals in an analytic framework, which synthesizes methods across established research disciplines. We outline the methods we are currently developing for the creation of a multimodal data corpus dedicated to the analysis and exploration of instrumental music performance from the perspective of embodied music cognition. This will enable the study of the multiple facets of instrumental music performance in great detail, as well as lead to the development of music creation techniques that take advantage of the cross-modal relationships and higher-level qualities emerging from the analysis of this multi-layered, multimodal corpus. The results of the pilot project suggest that qualitative analysis through stimulated recall is an efficient method for generating higher-level understandings of musical performance. Furthermore, the results indicate several directions for further development, regarding observational movement analysis, and computational analysis of coarticulation, chunking, and movement qualities in musical performance. We argue that the development of methods for combining qualitative and quantitative data are required to fully understand expressive musical performance, especially in a broader scenario in which arts, humanities, and science are increasingly entangled. The future work in the project will therefore entail an increasingly multimodal analysis, aiming to become as holistic as is music in performance.

show abstract

“…Reference [2] represented a critical survey on multimodal collaborative processing and retrieval of music information. The goal was to highlight how multimodal algorithms, working simultaneously on audio and video recordings, symbolic music scores, mid-level representations, motion and gestural data, etc., can help Music Computing applications.…”

Section: A Representationsmentioning

confidence: 99%

“…In conclusion, session Representations provided multifaceted views on the complex issue of digital music representation, describing the features of current formats [4] and proposing new ones [1], highlighting how a suitable representation can turn into an effective way to improve music information computing and retrieval [2], and proposing new forms of semantic representation of music items [3].…”

Section: A Representationsmentioning

confidence: 99%

Multilayer Music Representation and Processing: Key Advances and Emerging Trends

Avanzini

Ludovico

2019

2019 International Workshop on Multilayer Music Representation and Processing (MMRP)

Self Cite

View full text Add to dashboard Cite

This work represents the introduction to the proceedings of the 1 st International Workshop on Multilayer Music Representation and Processing (MMRP19) authored by the Program Co-Chairs. The idea is to explain the rationale behind such a scientific initiative, describe the methodological approach used in paper selection, and provide a short overview of the workshop's accepted works, trying to highlight the thread that runs through different contributions and approaches.

show abstract

Multimodal Music Information Processing and Retrieval: Survey and Future Challenges

Cited by 50 publications

References 43 publications

On the Adoption of Standard Encoding Formats to Ensure Interoperability of Music Digital Archives: The IEEE 1599 Format

On the Adoption of Standard Encoding Formats to Ensure Interoperability of Music Digital Archives: The IEEE 1599 Format

Method Development for Multimodal Data Corpus Analysis of Expressive Instrumental Music Performance

Multilayer Music Representation and Processing: Key Advances and Emerging Trends

Contact Info

Product

Resources

About