Continuous Speech Separation with Recurrent Selective Attention Network

Zhang, Yixuan; Chen, Zhuo; Wu, Jian; Yoshioka, Takuya; Wang, Peidong; Meng, Ziyang; Li, Jinyu

doi:10.1109/icassp43922.2022.9746394

Cited by 7 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A recent speech separation study [31] proposed the sourceaggregated SDR, which computes the mean over the SDR values of each output channel, this works as long as the target speaker is not absent in at least one output channel. Other studies [36], [37] used the mean squared error loss on the estimated mask or spectrogram to avoid the absent target speaker problem, while it remains unclear whether this is better as it is not directly optimizing the signal quality of the extracted speech waveform. In another study of the VAD-SE network [75], a target speaker voice activity detection (VAD) module and a speaker extraction network [75] are jointly trained.…”

Section: B Absent Target Speaker Speech Mixture With Audio Cuementioning

confidence: 99%

USEV: Universal Speaker Extraction With Visual Cue

Pan

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

A speaker extraction algorithm seeks to extract the target speaker's speech from a multi-talker speech mixture. The prior studies focus mostly on speaker extraction from a highly overlapped multi-talker speech mixture. However, the targetinterference speaker overlapping ratios could vary over a wide range from 0% to 100% in natural speech communication, furthermore, the target speaker could be absent in the speech mixture, the speech mixtures in such universal multi-talker scenarios are described as general speech mixtures. The speaker extraction algorithm requires an auxiliary reference, such as a video recording or a pre-recorded speech, to form top-down auditory attention on the target speaker. We advocate that a visual cue, i.e., lip movement, is more informative than an audio cue, i.e., pre-recorded speech, to serve as the auxiliary reference for speaker extraction in disentangling the target speaker from a general speech mixture. In this paper, we propose a universal speaker extraction network with a visual cue, that works for all multi-talker scenarios. In addition, we propose a scenarioaware differentiated loss function for network training, to balance the network performance over different target-interference speaker pairing scenarios. The experimental results show that our proposed method outperforms various competitive baselines for general speech mixtures in terms of signal fidelity.

show abstract

Section: B Absent Target Speaker Speech Mixture With Audio Cuementioning

confidence: 99%

USEV: Universal Speaker Extraction With Visual Cue

Pan

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…Many studies have been proposed to improve different aspects of the CSS framework [3,4,5,6,7,8,9,10]. We introduced a modulation factor based on segment overlap ratio to dynamically adjust the separation loss [3].…”

Section: Introductionmentioning

confidence: 99%

“…We introduced a modulation factor based on segment overlap ratio to dynamically adjust the separation loss [3]. In [4], a recurrent selective attention network is used to separate one speaker at a time. The work in [5] and [6] proposed new training criteria that generalizes PIT to capture long speech contexts.…”

Section: Introductionmentioning

confidence: 99%

Location-Based Training for Multi-Channel Talker-Independent Speaker Separation

Taherian¹,

Tan²,

Wang³

2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The performance of automatic speech recognition (ASR) systems severely degrades when multi-talker speech overlap occurs. In meeting environments, speech separation is typically performed to improve the robustness of ASR systems. Recently, location-based training (LBT) was proposed as a new training criterion for multi-channel talker-independent speaker separation. Assuming fixed array geometry, LBT outperforms widely-used permutation-invariant training in fully overlapped utterances and matched reverberant conditions. This paper extends LBT to conversational multi-channel speaker separation. We introduce multi-resolution LBT to estimate the complex spectrograms from low to high time and frequency resolutions. With multi-resolution LBT, convolutional kernels are assigned consistently based on speaker locations in physical space. Evaluation results show that multi-resolution LBT consistently outperforms other competitive methods on the recorded LibriCSS corpus.

show abstract