SubjQA: A Dataset for Subjectivity and Review Comprehension

Abstract

Subjectivity is the expression of internal opinions or beliefs which cannot be objectively observed or verified, and has been shown to be important for sentiment analysis and wordsense disambiguation. Furthermore, subjectivity is an important aspect of user-generated data. In spite of this, subjectivity has not been investigated in contexts where such data is widespread, such as in question answering (QA). We develop a new dataset which allows us to investigate this relationship. We find that subjectivity is an important feature in the case of QA, albeit with more intricate interactions between subjectivity and QA performance than found in previous work on sentiment analysis. For instance, a subjective question may or may not be associated with a subjective answer. We release an English QA dataset (SUBJQA) based on customer reviews, containing subjectivity annotations for questions and answer spans across 6 domains.

Unsupervised Evaluation for Question Answering with Transformers

“…We fine-tune BERT on either SQuAD v2.0 (Rajpurkar et al, 2018a) or SubjQA (Bjerva et al, 2020) before investigating the hidden representations. Since we analyse the similarity of hidden representations across answer span tokens, we only fine-tune BERT on answerable questions.…”

“…We experiment on two English-language QA datasets: SQuAD v2.0 (Rajpurkar et al, 2018a) and SubjQA (Bjerva et al, 2020). Since SQuAD v2.0 exclusively contains objective questions that belong to a single domain, Wikipedia, we contrast this with the more diverse SubjQA.…”

“…Since SQuAD v2.0 exclusively contains objective questions that belong to a single domain, Wikipedia, we contrast this with the more diverse SubjQA. Sub-jQA is a recently developed span-selection QA dataset that mainly consists of questions whose answer involves subjective opinions (Bjerva et al, 2020). Answer spans are extracted from review paragraphs that correspond to six different domains, namely books, electronics, groceries, movies, restaurants, tripadvisor.…”

“…We evaluate our method across two distinct QA datasets in arXiv:2010.03222v1 [cs.CL] 7 Oct 2020 English, covering 7 domains. We observe that the pattern can be used for automatic evaluation in both SQuAD v2.0 (Rajpurkar et al, 2018a) and SubjQA (Bjerva et al, 2020), a recently released dataset containing subjective questions and answers across several domains. We show that we can evaluate such models without any labelled test data, with an accuracy of 91.37% on SQuAD, and 80.7% accuracy on the more challenging and diverse SubjQA.…”

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

“…The earlier one was based on a Web crawl of questions and answers about products posed by users [167], and the more recent one (AmazonQA [101]) built upon it by cleaning up the data, and providing review snippets and (automatic) answerability annotation. Sub-jQA [28] is based on reviews from more sources than just Amazon, has manual answerability annotation and, importantly, is the first QA dataset to also include labels for subjectivity of answers.…”