Dialogue systems with audio context

Young, Tom; Pandelea, Vlad; Poria, Soujanya; Wang, Zhaoxia

doi:10.1016/j.neucom.2019.12.126

Cited by 39 publications

(17 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They also fine-tuned their DM using RL, but without including audio features. The work presented in [14] is probably the closest to our proposal. They investigate the inclusion of users' audio in an LSTM-based encoderdecoder network for response generation in open-domain dialogue -without reinforcement learning.…”

Section: Related Workmentioning

confidence: 62%

“…In contrast to their work, our approach is based on reinforcement learning, it is simpler in terms of implementation, it requires no further pretraining, and we apply it to task-oriented dialogue data for the first time. In addition, the audio representations used in [14] were trained without taking into account the word order in the turns, which could miss valuable audio information such as the prosody.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Audio Embeddings Help to Learn Better Dialogue Policies

Zorrilla

Torres

Cuayáhuitl

2021

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

Neural transformer architectures have gained a lot of interest for text-based dialogue management in the last few years. They have shown high learning capabilities for open domain dialogue with huge amounts of data and also for domain adaptation in task-oriented setups. But the potential benefits of exploiting the users' audio signal have rarely been explored in such frameworks. In this work, we combine text dialogue history representations generated by a GPT-2 model with audio embeddings obtained by the recently released Wav2Vec2 transformer model. We jointly fine-tune these models to learn dialogue policies via supervised learning and two policy gradient-based reinforcement learning algorithms. Our experimental results, using the DSTC2 dataset and a simulated user model capable of sampling audio turns, reveal that audio embeddings lead to overall higher task success (than without using audio embeddings) with statistically significant results across evaluation metrics and training algorithms.

show abstract

Section: Related Workmentioning

confidence: 62%

Section: Related Workmentioning

confidence: 99%

Audio Embeddings Help to Learn Better Dialogue Policies

Zorrilla

Torres

Cuayáhuitl

2021

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

show abstract

“…However, as could be observed in most of the surveyed works, they have not yet reached such a maturity level. Furthermore, trending topics in dialogue systems such as empathy [82], emotional information or sentiment analysis [83,84], and continual learning [85] have yet to be exploited by the community.…”

Section: Discussionmentioning

confidence: 99%

A Review of Plan-Based Approaches for Dialogue Management

Teixeira

Dragoni

2022

Cogn Comput

View full text Add to dashboard Cite

Dialogue systems deliver a more natural mean of communication between humans and machines when compared to traditional systems. Beyond input/output components that understand and generate natural language utterances, the core of a dialogue system is the dialogue manager. The aim of the dialogue manager is to mimic all cognitive aspects related to a natural conversation and it is responsible for identifying the current state of the dialogue and for deciding the next action to be taken by a dialogue system. Artificial intelligence (AI) planning is one of the techniques available in the literature for dialogue management. In a dialogue system, AI planning deals with the action selection problem by treating each utterance as an action and by choosing the actions that get closer to the dialogue goal. This work aims to provide a systematic literature review (SLR) that investigates recent contributions to plan-based dialogue management. This SLR aims at answering research questions concerning: (i) the types of AI planning exploited for dialogue management; (ii) the planning characteristics that justify its adoption in dialogue system; (iii) and, the challenges posed on the development of plan-based dialogue managers. The present SLR was performed by querying four scientific repositories, followed by a manual search on works from the most eminent authors in the field. Further works that were cited by the retrieved papers were also considered for inclusion. Our final corpus is composed of forty works, including only works published since 2014. The results indicate that AI planning is still an emerging strategy for dialogue management. Although AI planning can offer a strong contribution to dialogue systems, especially to those that require predictability, some relevant challenges might still limit its adoption. Our results contributed to discussions in the field and they highlight some research gaps to be addressed in future studies.

show abstract

“…Thus, some researchers argue that chatbots should also have such abilities to blend information from different modalities. There are some recent works trying to build multimodal dialogue systems (Le et al, 2019;Chauhan et al, 2019;Saha et al, 2020;Singla et al, 2020;Young et al, 2020), but these systems are still far from mature.…”

Section: Conclusion and Trendsmentioning

confidence: 99%

Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey

Ni¹,

Young²,

Pandelea³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Dialogue systems are a popular Natural Language Processing (NLP) task as it is promising in real-life applications. It is also a complicated task since many NLP tasks deserving study are involved. As a result, a multitude of novel works on this task are carried out, and most of them are deep learning based due to the outstanding performance. In this survey, we mainly focus on the deep learning based dialogue systems. We comprehensively review state-of-the-art research outcomes in dialogue systems and analyze them from two angles: model type and system type. Specifically, from the angle of model type, we discuss the principles, characteristics, and applications of different models that are widely used in dialogue systems. This will help researchers acquaint these models and see how they are applied in state-of-the-art frameworks, which is rather helpful when designing a new dialogue system. From the angle of system type, we discuss task-oriented and open-domain dialogue systems as two streams of research, providing insight into the hot topics related. Furthermore, we comprehensively review the evaluation methods and datasets for dialogue systems to pave the way for future research. Finally, some possible research trends are identified based on the recent research outcomes. To the best of our knowledge, this survey is the most comprehensive and up-to-date one at present in the area of dialogue systems and dialogue-related tasks, extensively covering the popular frameworks, topics, and datasets 1 . * Equal contribution ‡ Corresponding author 1 The frameworks, topics, and datasets discussed are originated from the extensive literature review of state-of-the-art research. We have tried our best to cover all but may still omit some works. Readers are welcome to provide suggestions regarding the omissions and mistakes in this article. We also intend to update this article with time as and when new approaches or definitions are proposed and used by the community Preprint. Under review.

show abstract

Dialogue systems with audio context

Cited by 39 publications

References 15 publications

Audio Embeddings Help to Learn Better Dialogue Policies

Audio Embeddings Help to Learn Better Dialogue Policies

A Review of Plan-Based Approaches for Dialogue Management

Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey

Contact Info

Product

Resources

About