Audio-Visual Scene Classification Using A Transfer Learning Based Joint Optimization Strategy

Chen, Chengxin; Wang, Meng; Zhang, Pengyuan

doi:10.48550/arxiv.2204.11420

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The audiovisual embedding is finally classified into target categories by a Dense Layer. From results shown in recent papers [8], [9], [11], we can see that the visual data contributes to the AVSC performance more significantly than the audio data. If we train our proposed AVSC system with an endto-end training process, it possibly causes an overfitting on the visual branch and reduces the role of the audio branch.…”

Section: Proposed Deep Learning Based Multimodal For Avsc Taskmentioning

confidence: 76%

“…Similar to the systems proposed for analyzing videos of human activities [5], [1], the state-of-the-art systems proposed for AVSC task also leveraged deep learning based models and presented joined audio-visual analysis. For instances, the proposed systems in [8], [9] used convolutional based models to extract audio embeddings from audio data and leveraged pre-trained deep learning models for extracting visual embeddings from visual data. Then, the audio embeddings and the visual embeddings are concatenated and fed into dense layers for classification.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Proposed Smart University Model as a Sustainable Living Lab for University Digital Transformation

Tuấn

Nguyen

Ngo

et al. 2020

2020 5th International Conference on Green Technology and Sustainable Development (GTSD)

View full text Add to dashboard Cite

In this paper, we present a deep learning based multimodal system for classifying daily life videos. To train the system, we propose a two-phase training strategy. In the first training phase (Phase I), we extract the audio and visual (image) data from the original video. We then train the audio data and the visual data with independent deep learning based models. After the training processes, we obtain audio embeddings and visual embeddings by extracting feature maps from the pretrained deep learning models. In the second training phase (Phase II), we train a fusion layer to combine the audio/visual embeddings and a dense layer to classify the combined embedding into target daily scenes. Our extensive experiments, which were conducted on the benchmark dataset of DCASE (IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events) 2021 Task 1B Development, achieved the best classification accuracy of 80.5%, 91.8%, and 95.3% with only audio data, with only visual data, both audio and visual data, respectively. The highest classification accuracy of 95.3% presents an improvement of 17.9% compared with DCASE baseline and shows very competitive to the state-of-the-art systems.

show abstract

Section: Proposed Deep Learning Based Multimodal For Avsc Taskmentioning

confidence: 76%