Subjective Annotation and Evaluation of Three Different Chatbots WOCHAT: Shared Task Report

Kong-Vega, Naomi; Shen, Mingxin; Wang, Mo; D’Haro, Luis Fernando

doi:10.1007/978-981-13-9443-0_32

Cited by 7 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another important challenge is the development of automatic evaluation protocols and metrics that can be used to reduce the current approach of asking humans to provide their subjective perception and detect mistakes (Deriu et al 2019; Kong‐Vega et al 2019). The limitation of having objective metrics directly focused for dialogue systems slow down the use of current state‐of‐the‐art artificial intelligence algorithms.…”

Section: Discussionmentioning

confidence: 99%

Considerations on creating conversational agents for multiple environments and users

Cebrián¹,

Martínez-Jiménez²,

Rodrı́guez³

et al. 2021

AI Magazine

Self Cite

View full text Add to dashboard Cite

Advances in artificial intelligence algorithms and expansion of straightforward cloud-based platforms have enabled the adoption of conversational assistants by both, medium and large companies, to facilitate interaction between clients and employees. The interactions are possible through the use of ubiquitous devices (e.g., Amazon Echo, Apple HomePod, Google Nest), virtual assistants (e.g., Apple Siri, Google Assistant, Samsung Bixby, or Microsoft Cortana), chat windows on the corporate website, or social network applications (e.g. Facebook Messenger, Telegram, Slack, WeChat).Creating a useful, personalized conversational agent that is also robust and popular is nonetheless challenging work. It requires picking the right algorithm, framework, and/or communication channel, but perhaps more importantly, consideration of the specific task, user needs, environment, available training data, budget, and a thoughtful design.In this paper, we will consider the elements necessary to create a conversational agent for different types of users, environments, and tasks. The elements will account for the limited amount of data available for specific tasks within a company and for non-English languages. We are confident that we can provide a useful resource for the new practitioner developing an agent. We can point out novice problems/traps to avoid, create consciousness that the development of the technology is achievable despite comprehensive and significant challenges, and raise awareness about different ethical issues that may be associated with this technology. We have compiled our experience with deploying conversational systems for daily use in multicultural, multilingual, and intergenerational settings. Additionally, we will give insight on how to scale the proposed solutions.

show abstract

Section: Discussionmentioning

confidence: 99%

Considerations on creating conversational agents for multiple environments and users

Cebrián¹,

Martínez-Jiménez²,

Rodrı́guez³

et al. 2021

AI Magazine

Self Cite

View full text Add to dashboard Cite

show abstract

“…Specifically, can we detect bots based on human response when data were collected under different settings and where the conversation topics differ? To address the out-of-domain research question, we use two additional datasets collected in the The Workshop on Chatbots and Conversational Agent Technologies (WOCHAT) (Kong-Vega et al, 2019). As part of a shared task, the workshop makes several bots available, has participants contribute new bots, and participants interact with the bots providing utterance-level feedback regarding their performance.…”

Section: Datasetsmentioning

confidence: 99%

Detecting Bot-Generated Text by Characterizing Linguistic Accommodation in Human-Bot Interactions

Bhatt¹,

Rios²

2021

Preprint

View full text Add to dashboard Cite

Language generation models' democratization benefits many domains, from answering health-related questions to enhancing education by providing AI-driven tutoring services. However, language generation models' democratization also makes it easier to generate human-like text at-scale for nefarious activities, from spreading misinformation to targeting specific groups with hate speech. Thus, it is essential to understand how people interact with bots and develop methods to detect bot-generated text. This paper shows that bot-generated text detection methods are more robust across datasets and models if we use information about how people respond to it rather than using the bot's text directly.We also analyze linguistic alignment, providing insight into differences between humanhuman and human-bot conversations.

show abstract

Reranking of Responses Using Transfer Learning for a Retrieval-Based Chatbot

Aksu

Chen

D’Haro

et al. 2021

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

Subjective Annotation and Evaluation of Three Different Chatbots WOCHAT: Shared Task Report

Cited by 7 publications

References 0 publications

Considerations on creating conversational agents for multiple environments and users

Considerations on creating conversational agents for multiple environments and users

Detecting Bot-Generated Text by Characterizing Linguistic Accommodation in Human-Bot Interactions

Reranking of Responses Using Transfer Learning for a Retrieval-Based Chatbot

Contact Info

Product

Resources

About