The TechQA Dataset

Abstract

We introduce TECHQA, a domain-adaptation question answering dataset for the technical support domain. The TECHQA corpus highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum, rather than questions generated specifically for a competition or a task. Second, it has a real-world size -600 training, 310 dev, and 490 evaluation question/answer pairs -thus reflecting the cost of creating large labeled datasets with actual data. Hence, TECHQA is meant to stimulate research in domain adaptation rather than as a resource to build QA systems from scratch. TECHQA was obtained by crawling the IB-MDeveloper and DeveloperWorks forums for questions with accepted answers provided in an IBM Technote-a technical document that addresses a specific technical issue. We also release a collection of the 801,998 Technotes available on the web as of April 4, 2019 as a companion resource that can be used to learn representations of the IT domain language.

Automatic Evaluation vs. User Preference in Neural Textual QuestionAnswering over COVID-19 Scientific Literature

“…In this sense the system trained on the SQuAD 2.0 dataset, which is not built with prospective users, achieves highest automatic metrics but falls behind when attending to user preference. There have been recent efforts for building QA datasets closer to real-world use cases (Choi et al, 2018;Reddy et al, 2019;Campos et al, 2020;Castelli et al, 2020). Among all of them, we show that the system trained also on the QuAC dataset is preferred by the users.…”

VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension

“…Domain Adaptation: Results on TechQA: Since VAULT has shown to be effective on NQ, we evaluate it on a new domain, TechQA. We compare it against a RoBERTa base model trained with the same hyper-parameters as (Castelli et al, 2020)except we use 11 epochs instead of 20. We chose base instead of large (as is used for the TechQA Results on TechQA are reported in Table 2.…”

“…Datasets: We experiment with two challenging "natural" MRC datasets: NQ (Kwiatkowski et al, 2019) and TechQA (Castelli et al, 2020). We provide a brief summary of the datasets and direct interested readers to the corresponding papers.…”

“…To evaluate the performance of VAULT, we select the new Natural Questions (NQ, Kwiatkowski et al, 2019) and TechQA (Castelli et al, 2020) datasets. NQ attempts to make Machine Reading Comprehension (MRC) more realistic by providing longer Wikipedia documents as contexts and real user search-engine queries as questions, and aims at avoiding observation bias: high lexical overlap between the question and the answer context which can happen frequently if the question is created after the user sees the paragraph (Rajpurkar et al, 2016(Rajpurkar et al, , 2018Karpukhin et al, 2020;Murdock et al, 2018).…”

Cheap and Good? Simple and Effective Data Augmentation for Low Resource Machine Reading

“…In particular, after transferring the approximate RC knowledge, the BERT-based MRC model's performance is improved by 3.46%, while the BERT-based document retriever's ( [2]) performance is improved by 4.33% (both in absolute F1 score) on TechQA, a realworld low-resource MRC dataset [2]. Further, our best model also outperforms the previous best document retrieval scores [2] by 15.12% in document retrieval accuracy (DRA) on TechQA.…”