A crowdsourcing method to develop virtual human conversational agents

Rossen, Brent; Lok, Benjamin

doi:10.1016/j.ijhcs.2011.11.004

Cited by 56 publications

(36 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the current medical domain, workers on common crowdsourcing platforms are unlikely to have the expertise required to take a patient's medical history in a natural way, so any data collected with this method would likely suffer quality issues and fail to generalize to real medical student dialogues. Rossen and Lok (2012) have developed an approach for collecting dialogue data for virtual patient systems, but their approach does not directly address the issue that even as the number of dialogues collected increases, there can remain a long 14 tail of relevant but infrequently asked questions.…”

Section: Related Workmentioning

confidence: 99%

Using Paraphrasing and Memory-Augmented Models to Combat Data Sparsity in Question Interpretation with a Virtual Patient Dialogue System

Jin

King

Hussein³

et al. 2018

Proceedings of the Thirteenth Workshop on Innovative Use of NLP For Building Educational Applications

View full text Add to dashboard Cite

When interpreting questions in a virtual patient dialogue system, one must inevitably tackle the challenge of a long tail of relatively infrequently asked questions. To make progress on this challenge, we investigate the use of paraphrasing for data augmentation and neural memory-based classification, finding that the two methods work best in combination. In particular, we find that the neural memory-based approach not only outperforms a straight CNN classifier on low frequency questions, but also takes better advantage of the augmented data created by paraphrasing, together yielding a nearly 10% absolute improvement in accuracy on the least frequently asked questions.

show abstract

Section: Related Workmentioning

confidence: 99%

Using Paraphrasing and Memory-Augmented Models to Combat Data Sparsity in Question Interpretation with a Virtual Patient Dialogue System

Jin

King

Hussein³

et al. 2018

Proceedings of the Thirteenth Workshop on Innovative Use of NLP For Building Educational Applications

View full text Add to dashboard Cite

show abstract

“…In the current medical domain, workers on common crowdsourcing platforms are unlikely to have the expertise required to take a patient's medical history in a natural way, so any data collected with this method would likely suffer quality issues and fail to generalize to real medical student dialogues. Rossen and Lok (2012) have developed an approach for collecting dialogue data for virtual patient systems, but their approach does not directly address the issue that even as the number of dialogues collected increases, there can remain a long tail of relevant but infrequently asked questions.…”

Section: Related Workmentioning

confidence: 99%

Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System

Jin

White

Jaffe

et al. 2017

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

For medical students, virtual patient dialogue systems can provide useful training opportunities without the cost of employing actors to portray standardized patients. This work utilizes word-and character-based convolutional neural networks (CNNs) for question identification in a virtual patient dialogue system, outperforming a strong word-and characterbased logistic regression baseline. While the CNNs perform well given sufficient training data, the best system performance is ultimately achieved by combining CNNs with a hand-crafted pattern matching system that is robust to label sparsity, providing a 10% boost in system accuracy and an error reduction of 47% as compared to the pattern-matching system alone.

show abstract

“…Thus far, we've focused on performance criteria that are largely assuming that answers each exist in isolation. In practice, some tasks require the system to maintain state or understand the implications of prior actions (e.g., when creating a virtual agent (Rossen and Lok, 2012;Borish et al, 2014;. This idea introduces a new measure of answer quality: consistency.…”

Section: Optimizing For Consistent Responsesmentioning

confidence: 99%

Architecting Real-Time Crowd-Powered Systems

Lasecki¹,

Homan²,

Bigham³

2014

HCj

View full text Add to dashboard Cite

Human computation allows computer systems to leverage human intelligence in computational processes. While it has primarily been used for tasks that are not time-sensitive, recent systems use crowdsourcing to get on-demand, real-time, and even interactive results. In this paper, we present techniques for building real-time crowdsourcing systems, and then discuss how and when to use them. Our goal is to provide system builders with the tools and insights they need to replicate the success of modern systems in order to further explore this new space.

show abstract

A crowdsourcing method to develop virtual human conversational agents

Cited by 56 publications

References 21 publications

Using Paraphrasing and Memory-Augmented Models to Combat Data Sparsity in Question Interpretation with a Virtual Patient Dialogue System

Using Paraphrasing and Memory-Augmented Models to Combat Data Sparsity in Question Interpretation with a Virtual Patient Dialogue System

Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System

Architecting Real-Time Crowd-Powered Systems

Contact Info

Product

Resources

About