VarArray: Array-Geometry-Agnostic Continuous Speech Separation

Yoshioka, Takuya; Wang, Xiaofei; Wang, Dongmei; Tang, Min; Zhu, Zirun; Chen, Zhuo; Kanda, Naoyuki

doi:10.48550/arxiv.2110.05745

Cited by 2 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The right diagram of Fig. 2 shows our proposed cross-channel layer, which was inspired by a transform-aggregate-concatenate model [12,13]. The idea is as follows.…”

Section: Picknet: Model With Cross-channel Layersmentioning

confidence: 99%

PickNet: Real-Time Channel Selection for Ad Hoc Microphone Arrays

Yoshioka¹,

Wang²,

Wang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper proposes PickNet, a neural network model for real-time channel selection for an ad hoc microphone array consisting of multiple recording devices like cell phones. Assuming at most one person to be vocally active at each time point, PickNet identifies the device that is spatially closest to the active person for each time frame by using a short spectral patch of just hundreds of milliseconds. The model is applied to every time frame, and the short time frame signals from the selected microphones are concatenated across the frames to produce an output signal. As the personal devices are usually held close to their owners, the output signal is expected to have higher signal-to-noise and direct-to-reverberation ratios on average than the input signals. Since PickNet utilizes only limited acoustic context at each time frame, the system using the proposed model works in real time and is robust to changes in acoustic conditions. Speech recognition-based evaluation was carried out by using real conversational recordings obtained with various smartphones. The proposed model yielded significant gains in word error rate with limited computational cost over systems using a block-online beamformer and a single distant microphone.

show abstract

“…The right diagram of Fig. 2 shows our proposed cross-channel layer, which was inspired by a transform-aggregate-concatenate model [12,13]. The idea is as follows.…”

Section: Picknet: Model With Cross-channel Layersmentioning

confidence: 99%

PickNet: Real-Time Channel Selection for Ad Hoc Microphone Arrays

Yoshioka¹,

Wang²,

Wang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Since real multi-channel recordings may be obtained from different devices with different microphone array geometries, it is desirable for a speech separation model to be able to deal with any array geometries without modification or retraining. Therefore, we make use of a recently proposed array-geometryagnostic model, called VarArray [12], in our investigation.…”

Section: Background Work 21 Vararray Modelmentioning

confidence: 99%

“…This is typically realized with window-based processing, where a speech separation neural network model is used to process each windowed signal. CSS can be performed with either single microphone [10,11] or a microphone array [5,12] with the latter being more effective in many cases.…”

Section: Introductionmentioning

confidence: 99%

“…However, this inevitably causes a mismatch between the training and deployment environments. An ASR-loss-based training approach [12,13,14,15] attempts to address this issue by using ground-truth transcriptions and an ASR model to define the loss function to minimize. Nonetheless, acquiring accurate transcriptions during speaker-overlapping periods is challenging even for humans, making this approach difficult to scale.…”

Section: Introductionmentioning

confidence: 99%

“…This framework allows us to fully leverage both a large amount of unlabeled real data and a small amount of high-quality transcribed real data. We use an array-geometry-agnostic multichannel separation model [12] to utilize multi-channel recordings from different microphone arrays. Unlike existing studies, we use real meeting recordings, i.e., Microsoft internal meetings [22] and the AMI corpus [23], and attempt to obtain insights that are relevant to real application scenarios.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation

Wang¹,

Wang²,

Kanda³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Existing multi-channel continuous speech separation (CSS) models are heavily dependent on supervised data -either simulated data which causes data mismatch between the training and real-data testing, or the real transcribed overlapping data, which is difficult to be acquired, hindering further improvements in the conversational/meeting transcription tasks. In this paper, we propose a three-stage training scheme for the CSS model that can leverage both supervised data and extra largescale unsupervised real-world conversational data. The scheme consists of two conventional training approaches-pre-training using simulated data and ASR-loss-based training using transcribed data-and a novel continuous semi-supervised training between the two, in which the CSS model is further trained by using real data based on the teacher-student learning framework. We apply this scheme to an array-geometry-agnostic CSS model, which can use the multi-channel data collected from any microphone array. Large-scale meeting transcription experiments are carried out on both Microsoft internal meeting data and the AMI meeting corpus. The steady improvement by each training stage has been observed, showing the effect of the proposed method that enables leveraging real conversational data for CSS model training.

show abstract

VarArray: Array-Geometry-Agnostic Continuous Speech Separation

Cited by 2 publications

References 23 publications

PickNet: Real-Time Channel Selection for Ad Hoc Microphone Arrays

PickNet: Real-Time Channel Selection for Ad Hoc Microphone Arrays

Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation

Contact Info

Product

Resources

About