CRQA: Crowd-Powered Real-Time Automatic Question Answering System

Abstract

Modern search engines have made dramatic progress in answering questions about facts, such as those that might be retrieved or directly inferred from a knowledge base. However, many other real user questions are more complex, such as requests for opinions, explanations, instructions or advice for a particular situation, and are still largely beyond the competence of the computer systems. As conversational agents become more popular, QA systems are increasingly expected to handle such complex questions, and to do so in (nearly) real-time, as the searcher is unlikely to wait longer than a minute or two for an answer. One way to overcome some of the challenges in complex question answering is crowdsourcing. We explore two ways crowdsourcing can assist a question answering system that operates in (near) real time: by providing answer validation, which could be used to filter or re-rank the candidate answers, and by creating the answer candidates directly. In this paper we present CRQA, a crowd-powered, near real-time automatic question answering system for complex informational tasks, that incorporates a crowdsourcing module for augmenting and validating the candidate answers. The crowd input, obtained in real-time, is integrated into CRQA via a learning-to-rank model, to select the final system answer. Our large-scale experiments, performed on a live stream of real users questions, show that even within a one minute time limit, CRQA can produce answers of high quality. The returned answers are judged to be significantly better compared to the automatic system alone, and even are often preferred to answers posted days later in the original community question answering site. Our findings can be useful for developing hybrid human-computer systems for automatic question answering and conversational agents.

A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks

“…Such multiple choice "tests" are relatively rare and most commonly employed when evaluating annotators based on known answers. A practical use case is when a model automatically predicts the top 4 or 5 answers to a question, randomizes these to avoid ranking bias, and then uses human computation to select the best answer (Yan, Kumar, & Ganesan, 2010;Rodriguez & Davis, 2011;Savenkov & Agichtein, 2016).…”

Software-Based Dialogue Systems: Survey, Taxonomy, and Challenges

“…The core of this area of application is the use and integration of conversational agents to achieve user's personal goals in daily life activities. In addition to general-purpose chatbots like crowd-powered Q&A agents serving as smart search engines [147], this domain includes different domain-specific application areas like tourism [116], restaurants and food [84], games [89], sports [151] and smart-home assistance [43]. • Commerce.…”

“…An example of this category is a self-adaptive agent designed to learn and adapt to new contexts and topics based on user interaction [71]. On the other hand, domaindependent agents include open or cross-domain (i.e., integrating multiple or several knowledge bases and domain data sources [147]) and closed-domain (i.e., focusing on a single, expert knowledge base [92]), which are the two main categories covered by Hussain et al [73] and Nuruzzaman and Hussain [118]. Generic and cross-domain chatbots typically require auxiliary NLP techniques to process and contextualize user input into a specific topic or domain, like co-reference resolution techniques [5].…”