Speaker Activity Driven Neural Speech Extraction

Delcroix, Marc; Žmolíková, Kateřina; Ochiai, Tsubasa; Kinoshita, Keisuke; Nakatani, Tomohiro

doi:10.1109/icassp39728.2021.9414998

Cited by 21 publications

(18 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In many scenarios, it may not be necessary to reconstruct all speakers from the mixture, instead, it suffices to extract a single target speaker. This task has been given numerous names in the literature, among which are target speaker extraction [14], [15], informed speaker extraction [16], or simply speaker extraction (SE) [17], [18]. In contrast to SS, SE systems do not suffer from the permutation ambiguity since there exists only one single output.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

New Insights on Target Speaker Extraction

Elminshawi¹,

Mack²,

Chakrabarty³

et al. 2022

Preprint

View full text Add to dashboard Cite

In recent years, researchers have become increasingly interested in speaker extraction (SE), which is the task of extracting the speech of a target speaker from a mixture of interfering speakers with the help of auxiliary information about the target speaker. Several forms of auxiliary information have been employed in single-channel SE, such as a speech snippet enrolled from the target speaker or visual information corresponding to the spoken utterance. Many SE studies have reported performance improvement compared to speaker separation (SS) methods with oracle selection, arguing that this is due to the use of auxiliary information. However, such works have not considered state-of-the-art SS methods that have shown impressive separation performance. In this paper, we revise and examine the role of the auxiliary information in SE. Specifically, we compare the performance of two SE systems (audio-based and video-based) with SS using a common framework that utilizes the state-of-the-art dual-path recurrent neural network as the main learning machine. In addition, we study how much the considered SE systems rely on the auxiliary information by analyzing the systems' output for random auxiliary signals.Experimental evaluation on various datasets suggests that the main purpose of the auxiliary information in the considered SE systems is only to specify the target speaker in the mixture and that it does not provide consistent extraction performance gain when compared to the uninformed SS system.

show abstract

Section: Introductionmentioning

confidence: 99%

“…Other methods have exploited multi-modal information, for example by utilizing both visual features of the target speaker as well as an enrolment utterance [29], [30]. Finally, brain signals [31] and speaker activity [15] have also been utilized as auxiliary signals for SE.…”

Section: Introductionmentioning

confidence: 99%

New Insights on Target Speaker Extraction

Elminshawi¹,

Mack²,

Chakrabarty³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The full separation attempts to estimate all sources in the mixture, which usually requires the knowledge of the number of sources. This arguably limits the practicality/flexibility of the separation (see the discussion in [18]- [20]). To alleviate, the recovery can be focused exclusively on the SOI, which is referred to as target speech extraction.…”

Section: Introductionmentioning

confidence: 99%

Target Speech Extraction: Independent Vector Extraction Guided by Supervised Speaker Identification

Málek

Janský

Koldovsky

et al. 2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

This manuscript proposes a novel robust procedure for the extraction of a speaker of interest (SOI) from a mixture of audio sources. The estimation of the SOI is performed via independent vector extraction (IVE). Since the blind IVE cannot distinguish the target source by itself, it is guided towards the SOI via frame-wise speaker identification based on deep learning. Still, an incorrect speaker can be extracted due to guidance failings, especially when processing challenging data. To identify such cases, we propose a criterion for nonintrusively assessing the estimated speaker. It utilizes the same model as the speaker identification, so no additional training is required. When incorrect extraction is detected, we propose a "deflation" step in which the incorrect source is subtracted from the mixture and, subsequently, another attempt to extract the SOI is performed. The process is repeated until successful extraction is achieved. The proposed procedure is experimentally tested on artificial and real-world datasets containing challenging phenomena: source movements, reverberation, transient noise, or microphone failures. The method is compared with state-of-theart blind algorithms as well as with current fully supervised deep learning-based methods.

show abstract

“…Once we obtain the semantic representations of concepts, we can identify the speech segments in the mixture related to them. Finally, we extract the target speech using the acoustic characteristics estimated from the identified segments in a manner similar to the activity driven extraction network (ADEnet) [8], which exploits speaker activity information for target speech extraction.…”

Section: Introductionmentioning

confidence: 99%

ConceptBeam: Concept Driven Target Speech Extraction

Ohishi,

Delcroix,

Ochiai

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a novel framework for target speech extraction based on semantic information, called ConceptBeam. Target speech extraction means extracting the speech of a target speaker in a mixture. Typical approaches have been exploiting properties of audio signals, such as harmonic structure and direction of arrival. In contrast, Con-ceptBeam tackles the problem with semantic clues. Specifically, we extract the speech of speakers speaking about a concept, i.e., a topic of interest, using a concept specifier such as an image or speech. Solving this novel problem would open the door to innovative applications such as listening systems that focus on a particular topic discussed in a conversation. Unlike keywords, concepts are abstract notions, making it challenging to directly represent a target concept. In our scheme, a concept is encoded as a semantic embedding by mapping the concept specifier to a shared embedding space. This modality-independent space can be built by means of deep metric learning using paired data consisting of images and their spoken captions. We use it to bridge modality-dependent information, i.e., the speech segments in the mixture, and the specified, modalityindependent concept. As a proof of our scheme, we performed experiments using a set of images associated with spoken captions. That is, we generated speech mixtures from these spoken captions and used the images or speech signals as the concept specifiers. We then extracted the target speech using the acoustic characteristics of the identified segments. We compare ConceptBeam with two methods: one based on keywords obtained from recognition systems and another based on sound source separation. We show that ConceptBeam clearly outperforms the baseline methods and effectively extracts speech based on the semantic representation.

show abstract

Speaker Activity Driven Neural Speech Extraction

Cited by 21 publications

References 28 publications

New Insights on Target Speaker Extraction

New Insights on Target Speaker Extraction

Target Speech Extraction: Independent Vector Extraction Guided by Supervised Speaker Identification

ConceptBeam: Concept Driven Target Speech Extraction

Contact Info

Product

Resources

About