ERANNs: Efficient residual audio neural networks for audio pattern recognition

Verbitskiy, Sergey; Berikov, Vladimir; Vyshegorodtsev, Viacheslav

doi:10.1016/j.patrec.2022.07.012

Cited by 39 publications

(17 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Efficient CNNs for Audio Tagging: Searching for efficient CNN architectures for AT and scaling models has been investigated before [1,3,2]. Efficient vision architectures, such as EfficientNets [8,9] and MobileNets [5,6,7] have shown to provide a good performance-complexity trade-off also in the audio domain [1,2,10].…”

Section: Related Workmentioning

confidence: 99%

“…Audio Tagging (AT) is the task of assigning one or multiple semantic labels to an audio clip. Until recently, CNNs have dominated the field of AT [1,2,3]. CNNs are a wellstudied and understood architecture for processing spectrograms and include well-suited inductive biases such as the locality bias, weight sharing and translation equivariance.…”

Section: Introductionmentioning

confidence: 99%

“…1. Crosses denote models based on Transformer architecture (Audio-MAE [19], HTS-AT [18], PaSST-S [17], PaSST-S-L [17], AST [16], KD-AST [10]) and circles denote models based on CNNs (PSLA [2], ERANN-1-6 [3], Wavegramlogmel-CNN [1], CNN14 [1], KD-CNN [10], MobileNets [7] -ours).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

Schmid¹,

Koutini²,

Widmer³

2022

Preprint

View full text Add to dashboard Cite

Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from highperforming yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-theart performance of .483 mAP on AudioSet. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

Schmid¹,

Koutini²,

Widmer³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Popular method to solve the audio classification task is usage of convolutional neural networks (CNN). In the way of representing the signal in the form of MFCCs it is possible to use 2D convolutional layers [Verbitskiy et al (2022)], this technique allows to train the model on the patterns of speech in terms of time, frequency and dynamic.…”

Section: Audio Emotionsmentioning

confidence: 99%

Detecting incongruity in the expression of emotions in short videos based on a multimodal approach

Laushkina

Smirnov

Medvedev

et al. 2022

CAP

View full text Add to dashboard Cite

Every day people face uncertainty, which is already an integral part of their lives. Uncertainty creates risks for various kinds of companies, in particular, the financial sector may incur losses due to various kinds of human errors. People turn to the opinion of experts who have special knowledge to eliminate this uncertainty. It is established that the expert shows insolvency if he uses incongruent manipulation techniques. In this article we propose a method that allows solving the problem of congruence estimation. The hypothesis that a person with a prepared speech and a person with a spontaneous speech will have a different level of congruence is also put forward and tested in this work. The similarity of emotional states of verbal and nonverbal channels is evaluated in our solution for determining congruence. Convolutional neural networks (CNN) were used to assess a person’s emotional state from video and audio, speeth-to-text to extract the text of the speaker’s speech, and a pre-trained BERT model for subsequent analysis of emotional color. Tests have shown that with the help of this development it is possible not only to distinguish the incongruence of a person, but also to point out the unnatural nature of his origin (to distinguish a simply incongruent person from a deepfake).

show abstract

“…A milestone for audio pattern recognition was the release of AudioSet [8], a dataset containing over 5000 h of audio recordings with 527 sound classes. Several CNN based models have been proposed for large scale audio classifi-cation [9][10][11][12][13]; however, pretrained audio neural network or PANN [14] is a VGGish [15] CNN-based model that achieves the state-of-the-art result for Audioset classification task. In the next section, we present the approach to detect the missing subtitle speech blocks.…”

Section: Audio Category Classificationmentioning

confidence: 99%

Language agnostic missing subtitle detection

Gupta¹,

Sharma²

2022

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Subtitles are a crucial component of Digital Entertainment Content (DEC such as movies and TV shows) localization. With ever increasing catalog (≈ 2M titles) and localization expansion (30+ languages), automated subtitle quality checks becomes paramount. Being a manual creation process, subtitles can have errors such as missing transcriptions, out-of-sync subtitle blocks with the audio and incorrect translations. Such erroneous subtitles result in an unpleasant viewing experience and impact the viewership. Moreover, manual correction is laborious, highly costly and requires expertise of audio and subtitle languages. A typical subtitle correction process consists of (1) linear watch of the movie, (2) identification of time stamps associated with erroneous subtitle blocks, and (3) correcting procedure. Among the three, time taken to watch the entire movie by a human expert is the most time consuming step. This paper discusses the problem of missing transcription, where the subtitle blocks corresponding to some speech segments in the DEC are non-existent. We present a solution to augment human correction process by automatically identifying the timings associated with the non-transcribed dialogues in a language agnostic manner. The correction step can then be performed by either human-in-the-loop mechanism or automatically using neural transcription (speech-to-text in same language) and translation (text-to-text in different languages) engines. Our method uses a language agnostic neural voice activity detector (VAD) and an audio classifier (AC) trained explicitly on DEC corpora for better generalization. The method consists of three steps: first, we use VAD to identify the timings associated with dialogues (predicted speech blocks). Second, we refine those timings using the AC module by removing the timings associated with the leading and trailing non-speech segments identified as speech by VAD. Finally, we compare the predicted dialogue timings to the dialogue timings present in the subtitle file (subtitle speech blocks) and flag the missing transcriptions. We empirically demonstrate that the proposed method (a) reduces incorrect predicted missing subtitle timings by 10%, (b) improves the predicted missing subtitle timings by 2.5%, (c) reduces false positive rate (FPR) of overextending the predicted timings by 77%, and (d) improves the predicted speech block-level precision by a 119% over VAD baseline on a human-annotated dataset of missing subtitle speech blocks.

show abstract

ERANNs: Efficient residual audio neural networks for audio pattern recognition

Cited by 39 publications

References 7 publications

Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

Detecting incongruity in the expression of emotions in short videos based on a multimodal approach

Language agnostic missing subtitle detection

Contact Info

Product

Resources

About