Crowdsourcing for (almost) Real-time Question Answering

Abstract

Modern search engines have made dramatic progress in the answering of many user's questions about facts, such as those that might be retrieved or directly inferred from a knowledge base. However, many other questions that real users ask are more complex, such as asking for opinions or advice for a particular situation, and are still largely beyond the competence of the computer systems. As conversational agents become more popular, QA systems are increasingly expected to handle such complex questions, and to do so in (nearly) real-time, as the searcher is unlikely to wait longer than a minute or two for an answer. One way to overcome some of the challenges in complex question answering is crowdsourcing. We explore two ways crowdsourcing can assist a question answering system that operates in (near) real time: by providing answer validation, which could be used to filter or re-rank the candidate answers, and by creating the answer candidates directly. Specifically, we focus on understanding the effects of time restrictions in the near real-time QA setting. Our experiments show that even within a one minute time limit, crowd workers can produce reliable ratings for up to three answer candidates, and generate answers that are better than an average automated system from the LiveQA 2015 shared task. Our findings can be useful for developing hybrid human-computer systems for automatic question answering and conversational agents.

CRQA: Crowd-Powered Real-Time Automatic Question Answering System

“…All the answers were randomly shuffled and rated on a scale from 1 (bad) to 4 (excellent) by workers hired on Amazon Mechanical Turk 8 . Existing research demonstrated, that such crowdsourced labels correlates well with the official ratings, provided by the professional NIST assessors (Savenkov, Weitzner, and Agichtein 2016). Each answer was labeled by 3 different workers, and we averaged the scores to get the final quality labels for the candidates.…”

“…Instead of immediately returning the answers, CRQA sends questions and top-7 ranked candidates to crowd workers and waits for the responses. We chose to give 7 answers based on the average number of rated answers in our preliminary studies (Savenkov, Weitzner, and Agichtein 2016). Since in TREC LiveQA systems had only 60 seconds to answer each Figure 3: User Interface for workers in our Crowd-Powered Question Answering system question, we start a timer when a question arrives, and the system waits to receive all worker contributions until the timer reaches 50 seconds to leave some time to generate the final answer.…”

Development of Answer Validation System Using Responders’ Attributes and Crowd Ranking

“…Afterwards, translation probabilities were computed via IBM and Hidden Markov Models to obtain the likelihood of an answer being the translation of the question. Savenkov et al (2016) presented a system that could be used to filter or re-rank the candidate answers by providing validation for the answers. They specifically focused on knowing the effect of time restrictions in the close real-time QA setting, thereby developing a way in which crowd will be able to create the answer candidates directly within a limited amount of time and also the way in which crowd will be able to rank sets of given answers to a question within a specified amount of time.…”

“…In this paper, we leverage on the fact that the performance of the crowd workers determines the quality of the result of a crowdsourcing task, and hence the need to develop an effective and reliable question answering system that is capable of validating and evaluating the answers provided by the crowd because of their varying reliability as established in past works (Hung et al, 2017;Savenkov et al, 2016). All these are important issues to be addressed in Artificial Intelligence.…”

A Dialogue System that Models User's Opinion Based on Information Content

“…Non-task oriented dialogues contribute to continuing conversations with users [1][2] and to building human social relationships [3]. There are two main methods for developing non-task oriented dialogue systems: a machine learning approach [4][5][6] and a rule-based one [7][8]. The former approach generally requires large corpus data.…”