MAE-AST: Masked Autoencoding Audio Spectrogram Transformer

Alan, Baade,; Puyuan, Peng,; Harwath, David

doi:10.21437/interspeech.2022-10961

Cited by 49 publications

(25 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of the ZRC series, a fair comparison equating architecture and dataset size would be necessary before claiming a definitive win. In addition, new models combining the two ideas like Masked AutoEncoders are emerging and need to be tested [86], [87].…”

Section: A Task 1: Acoustic Unit Discoverymentioning

confidence: 99%

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Dunbar

Hamilakis²,

Dupoux³

2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees. The contribution of the Zero Resource Speech Challenge series since 2015 has been to break down this long-term objective into four well-defined tasks-Acoustic Unit Discovery, Spoken Term Discovery, Discrete Resynthesis, and Spoken Language Modeling-and introduce associated metrics and benchmarks enabling model comparison and cumulative progress. We present an overview of the six editions of this challenge series since 2015, discuss the lessons learned, and outline the areas which need more work or give puzzling results.

show abstract

Section: A Task 1: Acoustic Unit Discoverymentioning

confidence: 99%

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Dunbar

Hamilakis²,

Dupoux³

2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

show abstract

“…Concretely, for Type I and Type II attacks, as the adversary does not involve in the pre-training phase, we utilize the public MAE 3 and CAE 4 as our target model. This aligns with the threat model that attackers can only get access to the released models.…”

Section: Experimental Settingsmentioning

confidence: 99%

“…Compared with contrastive learning which aims to align different augmented views of the same image, MIM learns from predicting properties of masked patches from unmasked parts. It plays as a milestone that bridges the gap between visual and linguistic self-supervised pre-training methods, and has quickly emerged variants in applications such as images [3,5], video [25,29], audio [4], and graph [23]. However, as an iconic method settling in another branch of SSL, the associated security risks caused by the mask-and-predict mechanism and novel architectures of MIM are still unexplored.…”

Section: Introductionmentioning

confidence: 99%

Backdoor Attacks in the Supply Chain of Masked Image Modeling

Shen¹,

He²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Masked image modeling (MIM) revolutionizes selfsupervised learning (SSL) for image pre-training. In contrast to previous dominating self-supervised methods, i.e., contrastive learning, MIM attains state-of-the-art performance by masking and reconstructing random patches of the input image. However, the associated security and privacy risks of this novel generative method are unexplored. In this paper, we perform the first security risk quantification of MIM through the lens of backdoor attacks. Different from previous work, we are the first to systematically threat modeling on SSL in every phase of the model supply chain, i.e., pre-training, release, and downstream phases. Our evaluation shows that models built with MIM are vulnerable to existing backdoor attacks in release and downstream phases and are compromised by our proposed method in pre-training phase. For instance, on CIFAR10, the attack success rate can reach 99.62%, 96.48%, and 98.89% in the downstream phase, release phase, and pre-training phase, respectively. We also take the first step to investigate the success factors of backdoor attacks in the pre-training phase and find the trigger number and trigger pattern play key roles in the success of backdoor attacks while trigger location has only tiny effects. In the end, our empirical study of the defense mechanisms across three detection-level on model supply chain phases indicates that different defenses are suitable for backdoor attacks in different phases. However, backdoor attacks in the release phase cannot be detected by all three detection-level methods, calling for more effective defenses in future research.

show abstract

“…It has been widely used in devices such as smart speakers and mobile phones for home safety or accessibility support [1]. Previous works have shown great progress on AEC using Convolutional Neural Network (CNN) [2,3,4,5] and Audio Spectrogram Transformer (AST) [6,7,8,9,10]. However, such models are usually computationally expensive and not suitable for edge devices (e.g., 86M model parameters for AST [6]).…”

Section: Introductionmentioning

confidence: 99%

Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints

Lin¹,

Tang²,

Kao³

et al. 2023

Preprint

View full text Add to dashboard Cite

Acoustic Event Classification (AEC) has been widely used in devices such as smart speakers and mobile phones for home safety or accessibility support [1]. As AEC models run on more and more devices with diverse computation resource constraints, it became increasingly expensive to develop models that are tuned to achieve optimal accuracy/computation trade-off for each given computation resource constraint. In this paper, we introduce a Once-For-All (OFA) Neural Architecture Search (NAS) framework for AEC. Specifically, we first train a weight-sharing supernet that supports different model architectures, followed by automatically searching for a model given specific computational resource constraints. Our experimental results showed that by just training once, the resulting model from NAS significantly outperforms both models trained individually from scratch and knowledge distillation (25.4% and 7.3% relative improvement). We also found that the benefit of weightsharing supernet training of ultra-small models comes not only from searching but from optimization.

show abstract

MAE-AST: Masked Autoencoding Audio Spectrogram Transformer

Cited by 49 publications

References 0 publications

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Backdoor Attacks in the Supply Chain of Masked Image Modeling

Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints

Contact Info

Product

Resources

About