A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI

Addlesee, Angus; Yu, Yanchao; Eshghi, Arash

doi:10.18653/v1/2020.coling-main.312

Cited by 14 publications

(14 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, reducing the overlap ratio by increasing the value of β also makes the duration of silence longer; thus, it is difficult to make the simulated mixture closer to a natural conversation by simply adjusting the value of β. Moreover, concatenating a zero vector in (6) to align the lengths of the long recordings causes unnaturalness of the conversation; various speakers speak at the beginning of the mixtures while only some of them speak at the end. In contrast, the proposed simulation protocol can generate more natural conversations by ordering each speaker's utterances to follow the statistics calculated from real conversational data.…”

Section: Conventional Simulation Methodmentioning

confidence: 99%

“…Speaker diarization is the task of identifying speech segments and their speakers from audio or video recordings; in other words, a task to identify "who spoke when" [1]. It is widely utilized in a variety of applications such as meeting transcription [2,3], conversational interaction analysis [4], content-based audio indexing [5], and conversational AI [6]. It also helps improve the accuracy of automatic speech recognition (ASR) in multi-speaker conversations [7].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization

Yamashita¹,

Horiguchi²,

Homma³

2022

Preprint

View full text Add to dashboard Cite

This paper investigates a method for simulating natural conversation in the model training of end-to-end neural diarization (EEND). Due to the lack of any annotated real conversational dataset, EEND is usually pretrained on a large-scale simulated conversational dataset first and then adapted to the target real dataset. Simulated datasets play an essential role in the training of EEND, but as yet there has been insufficient investigation into an optimal simulation method. We thus propose a method to simulate natural conversational speech. In contrast to conventional methods, which simply combine the speech of multiple speakers, our method takes turn-taking into account. We define four types of speaker transition and sequentially arrange them to simulate natural conversations. The dataset simulated using our method was found to be statistically similar to the real dataset in terms of the silence and overlap ratios. The experimental results on two-speaker diarization using the CALLHOME and CSJ datasets showed that the simulated dataset contributes to improving the performance of EEND.

show abstract

Section: Conventional Simulation Methodmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization

Yamashita¹,

Horiguchi²,

Homma³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Related to this, spontaneous human language production is often disfluent -it contains many fillers (um, er, uh etc), pauses, re-starts, and repairs. We therefore work on incremental language processing in understanding and generation, and the handling of disfluencies [57], often using the framework of Dynamic Syntax [2,19].…”

Section: Incremental Processingmentioning

confidence: 99%

“…This paper will largely focus on the first two types of system and the research problems and directions that they generate, as they are most closely related to research in Multi-Agent Systems. 2 In general we model each human conversation partner as an agent which has goals, plans, and preferences, and which can send signals (usually Natural Language speech or text 3 ) to other agents to convey its goals and request information, coordinate actions and plans etc.…”

Section: Introductionmentioning

confidence: 99%

Conversational AI for multi-agent communication in Natural Language

Lemon

2022

AIC

View full text Add to dashboard Cite

Research at the Interaction Lab focuses on human-agent communication using conversational Natural Language. The ultimate goal is to create systems where humans and AI agents (including embodied robots) can spontaneously form teams and coordinate shared tasks through the use of Natural Language conversation as a universal communication interface. This paper first introduces machine learning approaches to problems in conversational AI in general, where computational agents must coordinate with humans to solve tasks using conversational Natural Language. It also covers some of the practical systems developed in the Interaction Lab, ranging from speech interfaces on smart speakers to embodied robots interacting using visually grounded language. In several cases communication between multiple agents is addressed. The paper surveys the central research problems addressed here, the approaches developed, and our main results. Some key open research questions and directions are then discussed, leading towards a future vision of conversational, collaborative multi-agent systems.

show abstract

“…Since response time and online processing are the crucial factors in real-life settings, the demand for endto-end speaker diarization system integrated into ASR pipeline is growing. The performance of incremental (online) ASR and speaker diarization of the commercial ASR services are evaluated and compared in [228]. It is expected that the real-time and low latency aspect of speaker diarization will be more emphasized in the speaker diarization systems in the future since the performance of online diarization and online ASR still have much room for improvement.…”

Section: Conversational Aimentioning

confidence: 99%

A Review of Speaker Diarization: Recent Advances with Deep Learning

Park¹,

Kanda²,

Dimitriadis³

et al. 2021

Preprint

View full text Add to dashboard Cite

Speaker diarization is a task to label audio or video recordings with classes corresponding to speaker identity, or in short, a task to identify "who spoke when". In the early years, speaker diarization algorithms were developed for speech recognition on multispeaker audio recordings to enable speaker adaptive processing, but also gained its own value as a stand-alone application over time to provide speaker-specific meta information for downstream tasks such as audio retrieval. More recently, with the rise of deep learning technology that has been a driving force to revolutionary changes in research and practices across speech application domains in the past decade, more rapid advancements have been made for speaker diarization. In this paper, we review not only the historical development of speaker diarization technology but also the recent advancements in neural speaker diarization approaches. We also discuss how speaker diarization systems have been integrated with speech recognition applications and how the recent surge of deep learning is leading the way of jointly modeling these two components to be complementary to each other. By considering such exciting technical trends, we believe that it is a valuable contribution to the community to provide a survey work by consolidating the recent developments with neural methods and thus facilitating further progress towards a more efficient speaker diarization.

show abstract

A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI

Cited by 14 publications

References 25 publications

Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization

Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization

Conversational AI for multi-agent communication in Natural Language

A Review of Speaker Diarization: Recent Advances with Deep Learning

Contact Info

Product

Resources

About