Separate What You Describe: Language-Queried Audio Source Separation

Liu, Xubo; Liu, Haohe; Kong, Qiuqiang; Mei, Xinhao; Zhao, Jinzheng; Huang, Qiushi; Plumbley, Mark D.; Wang, Wenwu

doi:10.21437/interspeech.2022-10894

Cited by 29 publications

(12 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, the problem was extended to the extraction of arbitrary sounds from a mixture [66], [67], e.g., extracting the sound of a siren or a klaxon from a recording of a mixture of street sounds. We can use such systems as that introduced in Section IV to tackle these problems, where the clue can be a class label indicating the type of target sound [66], the enrollment audio of a similar target sound [67], a video of the sound source [9] or a text description of the target sound [68]. Target sound extraction may become an important technology to design, e.g., hearables or hearing aids that could filter out nuisances and emphasize important sounds in our surroundings, or audio visual scene analysis [9].…”

Section: Beyond Speechmentioning

confidence: 99%

Neural Target Speech Extraction: An Overview

Žmolíková¹,

Delcroix²,

Ochiai³

et al. 2023

Preprint

View full text Add to dashboard Cite

Humans can listen to a target speaker even in challenging acoustic conditions that have noise, reverberation, and interfering speakers. This phenomenon is known as the cocktail-party effect. For decades, researchers have focused on approaching the listening ability of humans. One critical issue is handling interfering speakers because the target and non-target speech signals share similar characteristics, complicating their discrimination. Target speech/speaker extraction (TSE) isolates the speech signal of a target speaker from a mixture of several speakers with or without noises and reverberations using clues that identify the speaker in the mixture. Such clues might be a spatial clue indicating the direction of the target speaker, a video of the speaker's lips, or a pre-recorded enrollment utterance from which their voice characteristics can be derived. TSE is an emerging field of research that has received increased attention in recent years because it offers a practical approach to the cocktailparty problem and involves such aspects of signal processing as audio, visual, array processing, and deep learning. This paper focuses on recent neural-based approaches and presents an in-depth overview of TSE. We guide readers through the different major approaches, emphasizing the similarities among frameworks and discussing potential future directions.

show abstract

Section: Beyond Speechmentioning

confidence: 99%

Neural Target Speech Extraction: An Overview

Žmolíková¹,

Delcroix²,

Ochiai³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Compared with flourishing research on VL multimodal learning, research on audio-language multimodal learning is limited due to the lack of AL datasets. Almost all AL tasks, such as automated audio captioning [10], [47], [48], languagebased audio retrieval [8], [9], text-to-audio generation [15], [16] and language-queried sound separation [19], rely on two main audio captioning datasets, AudioCaps [38] and Clotho [43]. AudioCaps contains about 50k audio clips sourced from AudioSet [1], the largest audio event dataset, and is annotated by humans.…”

Section: B Audio-language Datasetsmentioning

confidence: 99%

“…More recently, there has been a surge of interest in establishing a more profound comprehension of audio content by connecting audio and language. A number of audio-language (AL) multimodal learning tasks have been introduced, such as textto-audio retrieval [8], [9], automated audio captioning [10]- [12], audio question answering [13], [14], text-based sound generation [15]- [18], and text-based sound separation [19].…”

Section: Introductionmentioning

confidence: 99%

Automated audio captioning: an overview of recent progress and new challenges

Mei

Liu

Plumbley

et al. 2022

J AUDIO SPEECH MUSIC PROC.

Self Cite

View full text Add to dashboard Cite

Automated audio captioning is a cross-modal translation task that aims to generate natural language descriptions for given audio clips. This task has received increasing attention with the release of freely available datasets in recent years. The problem has been addressed predominantly with deep learning techniques. Numerous approaches have been proposed, such as investigating different neural network architectures, exploiting auxiliary information such as keywords or sentence information to guide caption generation, and employing different training strategies, which have greatly facilitated the development of this field. In this paper, we present a comprehensive review of the published contributions in automated audio captioning, from a variety of existing approaches to evaluation metrics and datasets. We also discuss open challenges and envisage possible future research directions.

show abstract

“…One more text-to-audio solution is AudioLDM, proposed by [9]. The authors emphasize higher generation quality with less expensive computing.…”

Section: Literature Reviewmentioning

confidence: 99%

Optimisation of conditions for deacetylation of chitin-containing raw materials

Павлова¹,

Trusova²

2021

FST

View full text Add to dashboard Cite

The paper describes the differences between chitosan and chitin, and reviews works by foreign scientists on obtaining chitosan from various raw materials. Methods of modifying chitosan and obtaining combined sorbents have been analysed. It has been studied whether chitosan is applicable in the technology of wines and alcoholic beverages as a sorbent. The purpose of the study was determining the optimal conditions of the deacetylation stage to obtain chitosan with the best sorption properties from Aspergillus niger biomass. A three-factor experiment has been carried out. It involved obtaining 27 samples of chitosan using sequential four-step acid-base hydrolysis under various conditions of the deacetylation stage. The deacetylation process was optimised under alkaline conditions depending on the alkali concentration, processing temperature, and exposure. For each of the samples obtained, the adsorption activity, specific surface area, and distribution coefficient in the sorbent–sorbate system have been determined. The degrees of deacetylation of all chitosan samples have been determined by potentiometric titration. The study has resulted in determining the optimal conditions for the deacetylation stage: processing temperature 110–130°C, sodium hydroxide concentration 27–36 g/dm3, exposure 45 to 65 minutes. The sample deacetylated at the temperature 120 °C, alkali concentration 30 g/dm3, and exposure 45 minutes has shown the best adsorption activity values: the adsorption activity for methyl orange 347.96 mg/g, the specific surface area of the sorbent samples 0.52·105 m2/g, the distribution coefficient in the sorbent–sorbate system 3.29·10-3 ml/g. This sample had the highest degree of deacetylation, 43.6%. The sample has been analysed using IR spectroscopy, and its main characteristic frequencies have been studied. It has been concluded that the sample obtained was equivalent to the reference chitosan

show abstract

Separate What You Describe: Language-Queried Audio Source Separation

Cited by 29 publications

References 48 publications

Neural Target Speech Extraction: An Overview

Neural Target Speech Extraction: An Overview

Automated audio captioning: an overview of recent progress and new challenges

Optimisation of conditions for deacetylation of chitin-containing raw materials

Contact Info

Product

Resources

About