Event-based video retrieval using audio

Qin, Jin; Schulam, Peter; Rawat, Shourabh; Burger, Susanne; Ding, Duo; Metze, Florian

doi:10.21437/interspeech.2012-556

Cited by 46 publications

(15 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, [4] use a siamese neural network framework to learn to encode semantically similar audio close together in the embedding space. [35] address multimedia event detection using only audio data, while [36] tackle near-duplicate video retrieval by audio retrieval. These are purely audio-based methods that are applied to video datasets, but without using visual information.…”

Section: Related Workmentioning

confidence: 99%

Audio Retrieval with Natural Language Queries

Oncescu¹,

Koepke²,

Henriques³

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the task of retrieving audio using free-form natural language queries. To study this problem, which has received limited attention in the existing literature, we introduce challenging new benchmarks for text-based audio retrieval using text annotations sourced from the AUDIOCAPS and CLOTHO datasets. We then employ these benchmarks to establish baselines for cross-modal audio retrieval, where we demonstrate the benefits of pre-training on diverse audio tasks. We hope that our benchmarks will inspire further research into cross-modal text-based audio retrieval with free-form text queries.

show abstract

Section: Related Workmentioning

confidence: 99%

Audio Retrieval with Natural Language Queries

Oncescu¹,

Koepke²,

Henriques³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Sound event detection (SED), in which the types of sound event are identified and their onset and offset in an audio recording are estimated, is one of the principal tasks in environmental sound analysis [1,2]. Recently, many works have addressed SED because it plays an important role in realizing various applications using artificial intelligence in sounds, e.g., automatic life logging, machine monitoring, automatic surveillance, media retrieval, and biomonitoring systems [3,4,5,6,7,8].…”

Section: Introductionmentioning

confidence: 99%

Impact of Sound Duration and Inactive Frames on Sound Event Detection Performance

Imoto

Mishima²,

Arai³

et al. 2021

Preprint

View full text Add to dashboard Cite

In many methods of sound event detection (SED), a segmented time frame is regarded as one data sample to model training. The durations of sound events greatly depend on the sound event class, e.g., the sound event "fan" has a long duration, whereas the sound event "mouse clicking" is instantaneous. Thus, the difference in the duration between sound event classes results in a serious data imbalance in SED. Moreover, most sound events tend to occur occasionally; therefore, there are many more inactive time frames of sound events than active frames. This also causes a severe data imbalance between active and inactive frames. In this paper, we investigate the impact of sound duration and inactive frames on SED performance by introducing four loss functions, such as simple reweighting loss, inverse frequency loss, asymmetric focal loss, and focal batch Tversky loss. Then, we provide insights into how we tackle this imbalance problem.

show abstract

“…Sound event detection (SED) is the task of detecting sound event labels and their onset/offset in an audio recording, where a sound event indicates a type of sound such as "people talking" and "bird singing" [1]. SED plays an important role in realizing various applications using artificial intelligence in sounds, such as automatic life-logging, machine monitoring, automatic surveillance, media retrieval, and biomonitoring systems [2][3][4][5][6][7][8].…”

Section: Introductionmentioning

confidence: 99%

Sound Event Detection Using Duration Robust Loss Function

Akiyama,

Imoto,

Tonami

et al. 2020

Preprint

View full text Add to dashboard Cite

Many methods of sound event detection (SED) based on machine learning regard a segmented time frame as one data sample to model training. However, the sound durations of sound events vary greatly depending on the sound event class, e.g., the sound event "fan" has a long time duration, while the sound event "mouse clicking" is instantaneous. The difference in the time duration between sound event classes thus causes a serious data imbalance problem in SED. In this paper, we propose a method for SED using a duration robust loss function, which can focus model training on sound events of short duration. In the proposed method, we focus on a relationship between the duration of the sound event and the ease/difficulty of model training. In particular, many sound events of long duration (e.g., sound event "fan") are stationary sounds, which have less variation in their acoustic features and their model training is easy. Meanwhile, some sound events of short duration (e.g., sound event "object impact") have more than one audio pattern, such as attack, decay, and release parts. We thus apply a classwise reweighting to the binary-cross entropy loss function depending on the ease/difficulty of model training. Evaluation experiments conducted using TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016 datasets show that the proposed method respectively improves the detection performance of sound events by 3.15 and 4.37 percentage points in macro-and micro-Fscores compared with a conventional method using the binary-cross entropy loss function.

show abstract

Event-based video retrieval using audio

Cited by 46 publications

References 7 publications

Audio Retrieval with Natural Language Queries

Audio Retrieval with Natural Language Queries

Impact of Sound Duration and Inactive Frames on Sound Event Detection Performance

Sound Event Detection Using Duration Robust Loss Function

Contact Info

Product

Resources

About