M-CTRL: A Continual Representation Learning Framework with Slowly Improving Past Pre-Trained Model

Choi, Jin-Seong; Lee, Jae Hong; Lee, Chae-Won; Chang, Joon‐Hyuk

doi:10.1109/icassp49357.2023.10096793

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2023

DOI: 10.1109/icassp49357.2023.10096793

|View full text |Cite

M-CTRL: A Continual Representation Learning Framework with Slowly Improving Past Pre-Trained Model

Jin-Seong Choi

Jae Hong Lee

Chae-Won Lee

et al.

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Article1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Masked Modeling Duo: Towards a Universal Audio Pre-Training Framework

Niizumi,

Takeuchi,

Ohishi

et al. 2024

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Self-supervised learning (SSL) using masked prediction has made great strides in general-purpose audio representation. This study proposes Masked Modeling Duo (M2D), an improved masked prediction SSL, which learns by predicting representations of masked input signals that serve as training signals. Unlike conventional methods, M2D obtains a training signal by encoding only the masked part, encouraging the two networks in M2D to model the input. While M2D improves general-purpose audio representations, a specialized representation is essential for real-world applications, such as in industrial and medical domains. The often confidential and proprietary data in such domains is typically limited in size and has a different distribution from that in pre-training datasets. Therefore, we propose M2D for X (M2D-X), which extends M2D to enable the pre-training of specialized representations for an application X. M2D-X learns from M2D and an additional task and inputs background noise. We make the additional task configurable to serve diverse applications, while the background noise helps learn on small data and forms a denoising task that makes representation robust. With these design choices, M2D-X should learn a representation specialized to serve various application needs. Our experiments confirmed that the representations for general-purpose audio, specialized for the highly competitive AudioSet and speech domain, and a small-data medical task achieve top-level performance, demonstrating the potential of using our models as a universal audio pre-training framework. Our code is available online for future studies.

show abstract

Masked Modeling Duo: Towards a Universal Audio Pre-Training Framework

Niizumi,

Takeuchi,

Ohishi

et al. 2024

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

M-CTRL: A Continual Representation Learning Framework with Slowly Improving Past Pre-Trained Model

Cited by 1 publication

References 9 publications

Masked Modeling Duo: Towards a Universal Audio Pre-Training Framework

Masked Modeling Duo: Towards a Universal Audio Pre-Training Framework

Contact Info

Product

Resources

About